Max level shown:
1 [ Jul 25 05:44:00 Enabled. ]
2 [ Jul 25 05:44:00 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]
3 [ Jul 25 05:44:00 Method "start" exited with status 0. ]
4 [ Jul 25 05:44:00 Rereading configuration. ]
5 [ Jul 25 05:44:00 No 'refresh' method defined. Treating as :true. ]
6 note: configured to log to "/dev/stdout"
72024-07-25T05:44:00.710ZINFOSledAgent: Ensuring contract reaper thread file = illumos-utils/src/running_zone.rs:110
82024-07-25T05:44:00.711ZINFOSledAgent (BootstrapAgentStartup): finding links ["net0", "net1"] file = sled-agent/src/bootstrap/pre_server.rs:255
92024-07-25T05:44:00.764ZINFOSledAgent (BootstrapAgentStartup): Starting mg-ddm service addr-objs = [AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }] file = sled-agent/src/bootstrap/pre_server.rs:262
102024-07-25T05:44:00.764ZINFOSledAgent (BootstrapAgentStartup): Importing mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:42 path = /opt/oxide/mg-ddm/pkg/ddm/manifest.xml
112024-07-25T05:44:00.880ZINFOSledAgent (BootstrapAgentStartup): Setting mg-ddm interfaces file = sled-agent/src/bootstrap/maghemite.rs:50 interfaces = ("net0/ll" "net1/ll")
122024-07-25T05:44:00.889ZINFOSledAgent (BootstrapAgentStartup): Enabling mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:56
132024-07-25T05:44:00.897ZINFOSledAgent (BootstrapAgentStartup): Ensuring zfs key directory exists file = sled-agent/src/bootstrap/pre_server.rs:272 path = /var/run/oxide/
142024-07-25T05:44:01.497ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fdb0:18c0:4d0d:9fb2::, width: 64 }
152024-07-25T05:44:01.640ZINFOSledAgent: Starting KeyManager file = sled-agent/src/long_running_tasks.rs:125
162024-07-25T05:44:01.640ZINFOSledAgent: Starting StorageManager file = sled-agent/src/long_running_tasks.rs:137
172024-07-25T05:44:01.640ZINFOSledAgent: Starting StorageMonitor file = sled-agent/src/long_running_tasks.rs:150
182024-07-25T05:44:01.640ZINFOSledAgent: Starting HardwareManager file = sled-agent/src/long_running_tasks.rs:172 nongimlet_observed_disks = [] sled_mode = Scrimlet { asic: SoftNpuZone }
192024-07-25T05:44:01.640ZINFOSledAgent (DumpSetup-worker): DumpSetup poll loop started. file = sled-agent/src/dump_setup.rs:520
202024-07-25T05:44:01.640ZINFOSledAgent (HardwareManager): Creating HardwareManager file = sled-hardware/src/illumos/mod.rs:718
212024-07-25T05:44:01.643ZINFOSledAgent (HardwareManager): Generated i86pc baseboard Pc { identifier: "buskin", model: "i86pc" } file = sled-hardware/src/illumos/mod.rs:625
222024-07-25T05:44:01.643ZWARNSledAgent (HardwareManager): Device is not a Gimlet (i86pc), proceeding with null hardware view file = sled-hardware/src/illumos/mod.rs:767
232024-07-25T05:44:01.643ZINFOSledAgent: Starting HardwareMonitor file = sled-agent/src/long_running_tasks.rs:186
242024-07-25T05:44:01.643ZINFOSledAgent (HardwareManager): Monitoring for hardware updates file = sled-hardware/src/illumos/mod.rs:826
252024-07-25T05:44:01.643ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/m2_0.vdev
262024-07-25T05:44:01.643ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = None
272024-07-25T05:44:01.643ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:404
282024-07-25T05:44:01.643ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
292024-07-25T05:44:01.643ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:127 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } path = /scratch/m2_0.vdev
302024-07-25T05:44:01.654ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_0.vdev file = sled-hardware/src/disk.rs:361
312024-07-25T05:44:01.654ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:372 id = 6d2aa15e-618e-402e-b368-cf27fd8def68 (zpool)
322024-07-25T05:44:01.750ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 6d2aa15e-618e-402e-b368-cf27fd8def68 (zpool), kind: Internal }
332024-07-25T05:44:01.960ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:404
342024-07-25T05:44:01.960ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/m2_1.vdev
352024-07-25T05:44:01.960ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
362024-07-25T05:44:01.960ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
372024-07-25T05:44:01.960ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:404
382024-07-25T05:44:01.960ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
392024-07-25T05:44:01.960ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:127 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } path = /scratch/m2_1.vdev
402024-07-25T05:44:01.969ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_1.vdev file = sled-hardware/src/disk.rs:361
412024-07-25T05:44:01.969ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:372 id = 75c788e1-45ef-4fcf-958d-db07331bef52 (zpool)
422024-07-25T05:44:02.056ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 75c788e1-45ef-4fcf-958d-db07331bef52 (zpool), kind: Internal }
432024-07-25T05:44:02.258ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_0.vdev
442024-07-25T05:44:02.258ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 6d2aa15e-618e-402e-b368-cf27fd8def68 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 75c788e1-45ef-4fcf-958d-db07331bef52 (zpool), kind: Internal } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
452024-07-25T05:44:02.258ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
462024-07-25T05:44:02.258ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:404
472024-07-25T05:44:02.258ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
482024-07-25T05:44:02.258ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
492024-07-25T05:44:02.259ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_1.vdev
502024-07-25T05:44:02.259ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 6d2aa15e-618e-402e-b368-cf27fd8def68 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 75c788e1-45ef-4fcf-958d-db07331bef52 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
512024-07-25T05:44:02.259ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
522024-07-25T05:44:02.259ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:404
532024-07-25T05:44:02.259ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
542024-07-25T05:44:02.259ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
552024-07-25T05:44:02.259ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_2.vdev
562024-07-25T05:44:02.259ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:404
572024-07-25T05:44:02.259ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 6d2aa15e-618e-402e-b368-cf27fd8def68 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 75c788e1-45ef-4fcf-958d-db07331bef52 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
582024-07-25T05:44:02.259ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
592024-07-25T05:44:02.259ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
602024-07-25T05:44:02.259ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
612024-07-25T05:44:02.259ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_3.vdev
622024-07-25T05:44:02.259ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:404
632024-07-25T05:44:02.259ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
642024-07-25T05:44:02.259ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 6d2aa15e-618e-402e-b368-cf27fd8def68 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 75c788e1-45ef-4fcf-958d-db07331bef52 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
652024-07-25T05:44:02.259ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
662024-07-25T05:44:02.259ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
672024-07-25T05:44:02.259ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_4.vdev
682024-07-25T05:44:02.259ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:404
692024-07-25T05:44:02.259ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 6d2aa15e-618e-402e-b368-cf27fd8def68 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 75c788e1-45ef-4fcf-958d-db07331bef52 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
702024-07-25T05:44:02.259ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
712024-07-25T05:44:02.259ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
722024-07-25T05:44:02.259ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
732024-07-25T05:44:02.259ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_5.vdev
742024-07-25T05:44:02.259ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:404
752024-07-25T05:44:02.259ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 6d2aa15e-618e-402e-b368-cf27fd8def68 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 75c788e1-45ef-4fcf-958d-db07331bef52 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
762024-07-25T05:44:02.259ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
772024-07-25T05:44:02.259ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
782024-07-25T05:44:02.259ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
792024-07-25T05:44:02.259ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_6.vdev
802024-07-25T05:44:02.259ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:404
812024-07-25T05:44:02.259ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
822024-07-25T05:44:02.259ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 6d2aa15e-618e-402e-b368-cf27fd8def68 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 75c788e1-45ef-4fcf-958d-db07331bef52 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
832024-07-25T05:44:02.259ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
842024-07-25T05:44:02.259ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
852024-07-25T05:44:02.259ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_7.vdev
862024-07-25T05:44:02.259ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:404
872024-07-25T05:44:02.259ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
882024-07-25T05:44:02.259ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 6d2aa15e-618e-402e-b368-cf27fd8def68 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 75c788e1-45ef-4fcf-958d-db07331bef52 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
892024-07-25T05:44:02.259ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
902024-07-25T05:44:02.259ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
912024-07-25T05:44:02.259ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_8.vdev
922024-07-25T05:44:02.259ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:404
932024-07-25T05:44:02.259ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
942024-07-25T05:44:02.259ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 6d2aa15e-618e-402e-b368-cf27fd8def68 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 75c788e1-45ef-4fcf-958d-db07331bef52 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
952024-07-25T05:44:02.259ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
962024-07-25T05:44:02.259ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
972024-07-25T05:44:02.259ZINFOSledAgent: Waiting for boot disk file = sled-agent/src/long_running_tasks.rs:96
982024-07-25T05:44:02.259ZINFOSledAgent: Found boot disk DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-agent/src/long_running_tasks.rs:98
992024-07-25T05:44:02.259ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 6d2aa15e-618e-402e-b368-cf27fd8def68 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 75c788e1-45ef-4fcf-958d-db07331bef52 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
1002024-07-25T05:44:02.259ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
1012024-07-25T05:44:02.259ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
1022024-07-25T05:44:02.259ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
1032024-07-25T05:44:02.259ZINFOSledAgent: Starting Bootstore file = sled-agent/src/long_running_tasks.rs:210
1042024-07-25T05:44:02.259ZINFOSledAgent (bootstore): No ledger in /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1052024-07-25T05:44:02.259ZINFOSledAgent (bootstore): No ledger in /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1062024-07-25T05:44:02.259ZINFOSledAgent (bootstore): No ledger found. Loading Fsm::State as Uninitialized file = bootstore/src/schemes/v0/storage.rs:83 peer_id = pc-buskin-i86pc
1072024-07-25T05:44:02.259ZINFOSledAgent (bootstore): No ledger in /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1082024-07-25T05:44:02.259ZINFOSledAgent (bootstore): No ledger in /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1092024-07-25T05:44:02.259ZINFOSledAgent (bootstore): No ledger found for network config file = bootstore/src/schemes/v0/storage.rs:146 peer_id = pc-buskin-i86pc
1102024-07-25T05:44:02.259ZINFOSledAgent: Starting Bootstore DDMD poller file = sled-agent/src/long_running_tasks.rs:215
1112024-07-25T05:44:02.259ZINFOSledAgent: Starting ZoneBundler related tasks file = sled-agent/src/long_running_tasks.rs:230
1122024-07-25T05:44:02.270ZINFOSledAgent (ServiceManager): Creating ServiceManager file = sled-agent/src/services.rs:719
1132024-07-25T05:44:02.270ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:292 next_cleanup = Instant { tv_sec: 6204, tv_nsec: 880097656 } time_to_next_cleanup = 599.999906668s
1142024-07-25T05:44:02.270ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3665
1152024-07-25T05:44:02.270ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
1162024-07-25T05:44:02.270ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/sled-agent-request.json file = common/src/ledger.rs:188
1172024-07-25T05:44:02.270ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/sled-agent-request.json file = common/src/ledger.rs:188
1182024-07-25T05:44:02.273ZINFOSledAgent (dropshot (BootstrapAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:205 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80
1192024-07-25T05:44:02.273ZINFOSledAgent (SledAgentSprocketsServer): Started listening file = sled-agent/src/bootstrap/sprockets_server.rs:44 local_addr = [fdb0:18c0:4d0d:9fb2::1]:12346
1202024-07-25T05:44:02.274ZINFOSledAgent (RSS): Injecting RSS configuration: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.1.164, last: 10.151.1.173, }, ), ], external_dns_ips: [ 10.151.1.164, 10.151.1.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.1.100, vlan_id: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.1.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, } file = sled-agent/src/rack_setup/service.rs:1014
1212024-07-25T05:44:02.274ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
1222024-07-25T05:44:02.274ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
1232024-07-25T05:44:02.274ZINFOSledAgent (RSS): No ledger in /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1242024-07-25T05:44:02.274ZINFOSledAgent (RSS): No ledger in /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1252024-07-25T05:44:02.274ZINFOSledAgent (RSS): RSS configuration has not been fully applied yet file = sled-agent/src/rack_setup/service.rs:1074
1262024-07-25T05:44:02.274ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
1272024-07-25T05:44:02.274ZINFOSledAgent (RSS): No ledger in /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/rss-sled-plan.json file = common/src/ledger.rs:188
1282024-07-25T05:44:02.274ZINFOSledAgent (RSS): No ledger in /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/rss-sled-plan.json file = common/src/ledger.rs:188
1292024-07-25T05:44:02.274ZINFOSledAgent (RSS): Creating new allocation plan file = sled-agent/src/rack_setup/service.rs:1115
1302024-07-25T05:44:02.274ZINFOSledAgent (RSS): Serializing plan file = sled-agent/src/rack_setup/plan/sled.rs:156
1312024-07-25T05:44:02.274ZINFOSledAgent (RSS): Creating plan for the sled at fdb0:18c0:4d0d:9fb2::1 file = sled-agent/src/rack_setup/plan/sled.rs:129
1322024-07-25T05:44:02.274ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
1332024-07-25T05:44:02.274ZINFOSledAgent (RSS): Writing ledger to /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1342024-07-25T05:44:02.274ZINFOSledAgent (RSS): Writing ledger to /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1352024-07-25T05:44:02.274ZINFOSledAgent (RSS): Sled plan written to storage file = sled-agent/src/rack_setup/plan/sled.rs:176
1362024-07-25T05:44:02.274ZINFOSledAgent (RSS): Writing Rack Network Configuration to bootstore file = sled-agent/src/rack_setup/service.rs:1148
1372024-07-25T05:44:02.274ZINFOSledAgent (bootstore): Attempting to update network config with generation: 1, current_generation: 0 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-buskin-i86pc
1382024-07-25T05:44:02.274ZINFOSledAgent (bootstore): Writing ledger to /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
1392024-07-25T05:44:02.275ZINFOSledAgent (bootstore): Writing ledger to /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
1402024-07-25T05:44:02.275ZINFOSledAgent (bootstore): Broadcasting network config with generation 1 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-buskin-i86pc
1412024-07-25T05:44:02.275ZINFOSledAgent (BootstrapAgentRssHandler): Received initialization request from RSS file = sled-agent/src/bootstrap/rss_handle.rs:217 request = StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 8fa4bc78-1026-41f0-b09d-c67224071376, rack_id: b4c6410d-62e1-4369-a8c0-f8e6399dd808, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } target_sled = [fdb0:18c0:4d0d:9fb2::1]:12346
1422024-07-25T05:44:02.275ZINFOSledAgent (SledAgentSprocketsServer): Accepted connection file = sled-agent/src/bootstrap/sprockets_server.rs:66 remote_addr = [fdb0:18c0:4d0d:9fb2::1]:57030
1432024-07-25T05:44:02.275ZINFOSledAgent (SledAgentMain): Loading Sled Agent: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 8fa4bc78-1026-41f0-b09d-c67224071376, rack_id: b4c6410d-62e1-4369-a8c0-f8e6399dd808, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } file = sled-agent/src/bootstrap/server.rs:359
1442024-07-25T05:44:02.275ZINFOSledAgent (SledAgentMain): KeyManager: using hardcoded secret retriever file = sled-agent/src/bootstrap/server.rs:374
1452024-07-25T05:44:02.275ZINFOSledAgent: setting up sled agent server file = sled-agent/src/server.rs:44
1462024-07-25T05:44:02.275ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
1472024-07-25T05:44:02.275ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:101::, width: 64 }
1482024-07-25T05:44:02.275ZINFOSledAgent (StorageManager): Received KeyManagerReady file = sled-storage/src/manager.rs:404
1492024-07-25T05:44:02.275ZINFOSledAgent (StorageManager): Loading storage config file = sled-storage/src/manager.rs:572
1502024-07-25T05:44:02.275ZINFOSledAgent (StorageManager): No ledger in /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_ledger
1512024-07-25T05:44:02.275ZINFOSledAgent (StorageManager): No ledger in /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_ledger
1522024-07-25T05:44:02.275ZINFOSledAgent (StorageManager): No ledger of physical disks exists file = sled-storage/src/manager.rs:561
1532024-07-25T05:44:02.275ZINFOSledAgent (StorageManager): KeyManager ready, but no ledger detected file = sled-storage/src/manager.rs:589
1542024-07-25T05:44:02.275ZINFOSledAgent (StorageManager): automatically managing disks file = sled-storage/src/manager.rs:393
1552024-07-25T05:44:02.276ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:374
1562024-07-25T05:44:02.276ZINFOSledAgent (StorageManager): Successfully synchronized disks without error file = sled-storage/src/manager.rs:479 result = DisksManagementResult { status: [] }
1572024-07-25T05:44:02.277ZINFOSledAgent (ServiceManager): Enabling switch zone (new) file = sled-agent/src/services.rs:3899
1582024-07-25T05:44:02.277ZINFOSledAgent (ServiceManager): Starting switch zone file = sled-agent/src/services.rs:4272
1592024-07-25T05:44:02.308ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
1602024-07-25T05:44:02.328ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:316
1612024-07-25T05:44:02.345ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:354
1622024-07-25T05:44:02.392ZINFOSledAgent: SledAgent::new(..) starting file = sled-agent/src/sled_agent.rs:365 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
1632024-07-25T05:44:02.393ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
1642024-07-25T05:44:02.393ZINFOSledAgent: Requested swap device of size 64 GiB file = sled-agent/src/sled_agent.rs:377 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
1652024-07-25T05:44:02.393ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1662024-07-25T05:44:02.396ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 114.258176ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "32c6bba2-2f6a-4a9a-ac7a-4e82ed3164a6", "content-length": "124", "date": "Thu, 25 Jul 2024 05:44:01 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "32c6bba2-2f6a-4a9a-ac7a-4e82ed3164a6" } file = clients/ddm-admin-client/src/lib.rs:177
1672024-07-25T05:44:02.407ZINFOSledAgent: attempting to create encrypted zvol: name="oxi_6d2aa15e-618e-402e-b368-cf27fd8def68/swap", size_gb=64 file = sled-agent/src/swap_device.rs:153
1682024-07-25T05:44:02.431ZINFOSledAgent: successfully created encrypted zvol: name="oxi_6d2aa15e-618e-402e-b368-cf27fd8def68/swap", size_gb=64 file = sled-agent/src/swap_device.rs:249
1692024-07-25T05:44:02.431ZINFOSledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_6d2aa15e-618e-402e-b368-cf27fd8def68/swap" file = sled-agent/src/swap_device.rs:96
1702024-07-25T05:44:02.437ZINFOSledAgent: Mounting backing filesystems file = sled-agent/src/sled_agent.rs:392 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
1712024-07-25T05:44:02.437ZINFOSledAgent (BackingFs): Processing fmd file = sled-agent/src/backing_fs.rs:126
1722024-07-25T05:44:02.437ZINFOSledAgent (BackingFs): Ensuring dataset oxi_6d2aa15e-618e-402e-b368-cf27fd8def68/backing/fmd file = sled-agent/src/backing_fs.rs:136
1732024-07-25T05:44:02.511ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1742024-07-25T05:44:02.511ZINFOSledAgent (BackingFs): Stopping service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:170
1752024-07-25T05:44:02.514ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 100.741553ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "4aaa740d-1eec-4bce-8079-c1434d2c2a92", "content-length": "124", "date": "Thu, 25 Jul 2024 05:44:02 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "4aaa740d-1eec-4bce-8079-c1434d2c2a92" } file = clients/ddm-admin-client/src/lib.rs:177
1762024-07-25T05:44:02.616ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1772024-07-25T05:44:02.616ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 91.514047ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "49e761db-b21b-4cdb-841d-4f56eac50283", "content-length": "124", "date": "Thu, 25 Jul 2024 05:44:02 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "49e761db-b21b-4cdb-841d-4f56eac50283" } file = clients/ddm-admin-client/src/lib.rs:177
1782024-07-25T05:44:02.708ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1792024-07-25T05:44:02.709ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 214.652667ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "ef7941a1-220e-4365-a1a5-9a0df00b48c0", "content-length": "124", "date": "Thu, 25 Jul 2024 05:44:02 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "ef7941a1-220e-4365-a1a5-9a0df00b48c0" } file = clients/ddm-admin-client/src/lib.rs:177
1802024-07-25T05:44:02.731ZINFOSledAgent (BackingFs): Mounting oxi_6d2aa15e-618e-402e-b368-cf27fd8def68/backing/fmd on /var/fm/fmd file = sled-agent/src/backing_fs.rs:178
1812024-07-25T05:44:02.744ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/rsrc file = sled-agent/src/backing_fs.rs:186
1822024-07-25T05:44:02.744ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/ckpt file = sled-agent/src/backing_fs.rs:186
1832024-07-25T05:44:02.744ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/xprt file = sled-agent/src/backing_fs.rs:186
1842024-07-25T05:44:02.744ZINFOSledAgent (BackingFs): Starting service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:192
1852024-07-25T05:44:02.925ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1862024-07-25T05:44:02.926ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 124.25303ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "a088797f-8bcf-40b8-ac5b-4f8eb1851afe", "content-length": "124", "date": "Thu, 25 Jul 2024 05:44:02 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "a088797f-8bcf-40b8-ac5b-4f8eb1851afe" } file = clients/ddm-admin-client/src/lib.rs:177
1872024-07-25T05:44:03.051ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1882024-07-25T05:44:03.051ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 346.946173ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "d36e60cc-f22d-45e4-9d39-ec187268c576", "content-length": "124", "date": "Thu, 25 Jul 2024 05:44:02 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "d36e60cc-f22d-45e4-9d39-ec187268c576" } file = clients/ddm-admin-client/src/lib.rs:177
1892024-07-25T05:44:03.399ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1902024-07-25T05:44:03.400ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 275.491254ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "e9776b26-434b-4ed0-977c-e01509241d2d", "content-length": "124", "date": "Thu, 25 Jul 2024 05:44:02 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "e9776b26-434b-4ed0-977c-e01509241d2d" } file = clients/ddm-admin-client/src/lib.rs:177
1912024-07-25T05:44:03.467ZINFOSledAgent: using '[AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }]' as data links for xde driver file = illumos-utils/src/opte/illumos.rs:90 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
1922024-07-25T05:44:03.467ZINFOSledAgent (dropshot): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:205 local_addr = [fd00:1122:3344:101::1]:57429 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
1932024-07-25T05:44:03.467ZINFOSledAgent (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
1942024-07-25T05:44:03.468ZINFOSledAgent (producer-server): starting oximeter metric producer server address = [fd00:1122:3344:101::1]:57429 file = oximeter/producer/src/lib.rs:283 interval = 30s producer_id = 8fa4bc78-1026-41f0-b09d-c67224071376 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
1952024-07-25T05:44:03.474ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 342.575265ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
1962024-07-25T05:44:03.514ZINFOSledAgent (VmmReservoirManager): 50% of 102952026112 physical ram = 51476013056 bytes) file = sled-agent/src/vmm_reservoir.rs:267 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
1972024-07-25T05:44:03.514ZINFOSledAgent (VmmReservoirManager): Setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:275 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
1982024-07-25T05:44:03.514ZINFOSledAgent (ServiceManager): sled agent started file = sled-agent/src/services.rs:902 underlay_address = fd00:1122:3344:101::1
1992024-07-25T05:44:03.514ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
2002024-07-25T05:44:03.515ZINFOSledAgent (dropshot (SledAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:205 local_addr = [fd00:1122:3344:101::1]:12345
2012024-07-25T05:44:03.515ZINFOSledAgent (SledAgentMain): Sled Agent loaded; recording configuration file = sled-agent/src/bootstrap/server.rs:429
2022024-07-25T05:44:03.515ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
2032024-07-25T05:44:03.515ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2042024-07-25T05:44:03.515ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2052024-07-25T05:44:03.515ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/8fa4bc78-1026-41f0-b09d-c67224071376): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
2062024-07-25T05:44:03.515ZINFOSledAgent (HardwareMonitor): Sled Agent Started file = sled-agent/src/hardware_monitor.rs:138
2072024-07-25T05:44:03.515ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = Some((fd00:1122:3344:101::2, Some(RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56 }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [PortConfigV2 { routes: [RouteConfig { destination: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), nexthop: 10.151.1.100, vlan_id: None }], addresses: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false }], bgp: [], bfd: [] })))
2082024-07-25T05:44:03.515ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3665
2092024-07-25T05:44:03.515ZINFOSledAgent (SledAgentSprocketsServer): Connection closed file = sled-agent/src/bootstrap/sprockets_server.rs:73 remote_addr = [fdb0:18c0:4d0d:9fb2::1]:57030
2102024-07-25T05:44:03.516ZINFOSledAgent (BootstrapAgentRssHandler): Peer agent initialized file = sled-agent/src/bootstrap/rss_handle.rs:108 peer = [fdb0:18c0:4d0d:9fb2::1]:12346
2112024-07-25T05:44:03.516ZINFOSledAgent (BootstrapAgentRssHandler): Initialized sled agent file = sled-agent/src/bootstrap/rss_handle.rs:232 target_sled = [fdb0:18c0:4d0d:9fb2::1]:12346
2122024-07-25T05:44:03.701ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
2132024-07-25T05:44:03.702ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
2142024-07-25T05:44:03.702ZINFOSledAgent (RSS): No ledger in /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/rss-service-plan-v3.json file = common/src/ledger.rs:188
2152024-07-25T05:44:03.702ZINFOSledAgent (RSS): No ledger in /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/rss-service-plan-v3.json file = common/src/ledger.rs:188
2162024-07-25T05:44:03.702ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
2172024-07-25T05:44:03.702ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
2182024-07-25T05:44:03.738ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:63451
2192024-07-25T05:44:03.738ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
2202024-07-25T05:44:03.738ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 146 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:63451 req_id = 045b0d14-d814-4535-a4eb-6b7364b324f0 response_code = 200 uri = /inventory
2212024-07-25T05:44:03.770ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:49309
2222024-07-25T05:44:03.770ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 39 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:49309 req_id = 9e90c4bc-0876-4e78-b043-be37c153c455 response_code = 200 uri = /sled-role
2232024-07-25T05:44:03.772ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
2242024-07-25T05:44:03.772ZINFOSledAgent (RSS): Writing ledger to /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/.rss-service-plan-v3.json.tmp file = common/src/ledger.rs:199
2252024-07-25T05:44:03.772ZINFOSledAgent (RSS): Writing ledger to /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/.rss-service-plan-v3.json.tmp file = common/src/ledger.rs:199
2262024-07-25T05:44:03.772ZINFOSledAgent (RSS): Service plan written to storage file = sled-agent/src/rack_setup/plan/service.rs:832
2272024-07-25T05:44:03.802ZINFOSledAgent (RSS): attempting to set up sled's storage: OmicronPhysicalDisksConfig { disks: [OmicronPhysicalDiskConfig { id: 489d3050-bf57-46d2-bf46-7cd957436e3d, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, pool_id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool) }, OmicronPhysicalDiskConfig { id: 4edc2975-bd7a-4aad-80f3-9409995355d1, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, pool_id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool) }, OmicronPhysicalDiskConfig { id: e2770ac5-7ae7-4600-bf20-0849191c7d7c, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, pool_id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool) }, OmicronPhysicalDiskConfig { id: 7b9e4bc9-bcfc-41a0-be73-09102123a06a, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, pool_id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool) }, OmicronPhysicalDiskConfig { id: d8e66d81-f03a-4b7f-94b4-e281f422c437, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, pool_id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool) }, OmicronPhysicalDiskConfig { id: 282f71da-0fb9-4a90-9caa-dd76e77730ca, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, pool_id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool) }, OmicronPhysicalDiskConfig { id: 0cd36a66-88d7-45b4-82f8-717f3515dbd9, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, pool_id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool) }, OmicronPhysicalDiskConfig { id: 6e68f196-dc3d-4e49-9aac-95ce41525cb3, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, pool_id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool) }, OmicronPhysicalDiskConfig { id: 3c83f3c1-50fb-429e-b2d4-58a398af121f, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, pool_id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool) }], generation: Generation(1) } file = sled-agent/src/rack_setup/service.rs:351 sled_address = [fd00:1122:3344:101::1]:12345
2282024-07-25T05:44:03.802ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:47856
2292024-07-25T05:44:03.803ZINFOSledAgent: physical disks ensure file = sled-agent/src/sled_agent.rs:824 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
2302024-07-25T05:44:03.803ZINFOSledAgent (StorageManager): Received OmicronPhysicalDisksEnsure { config: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: 489d3050-bf57-46d2-bf46-7cd957436e3d, pool_id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: 4edc2975-bd7a-4aad-80f3-9409995355d1, pool_id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: e2770ac5-7ae7-4600-bf20-0849191c7d7c, pool_id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: 7b9e4bc9-bcfc-41a0-be73-09102123a06a, pool_id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: d8e66d81-f03a-4b7f-94b4-e281f422c437, pool_id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: 282f71da-0fb9-4a90-9caa-dd76e77730ca, pool_id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 0cd36a66-88d7-45b4-82f8-717f3515dbd9, pool_id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: 6e68f196-dc3d-4e49-9aac-95ce41525cb3, pool_id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: 3c83f3c1-50fb-429e-b2d4-58a398af121f, pool_id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool) }] }, tx: ... } file = sled-storage/src/manager.rs:404
2312024-07-25T05:44:03.803ZINFOSledAgent (StorageManager): No ledger in /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2322024-07-25T05:44:03.803ZINFOSledAgent (StorageManager): No ledger in /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2332024-07-25T05:44:03.803ZINFOSledAgent (StorageManager): No previously-stored 'requested disks', creating new ledger file = sled-storage/src/manager.rs:643 request = omicron_physical_disks_ensure
2342024-07-25T05:44:03.803ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:374
2352024-07-25T05:44:03.803ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:414
2362024-07-25T05:44:03.803ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
2372024-07-25T05:44:03.803ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:127 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } path = /scratch/u2_0.vdev
2382024-07-25T05:44:03.906ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 709.360328ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
2392024-07-25T05:44:03.920ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_0.vdev file = sled-hardware/src/disk.rs:361
2402024-07-25T05:44:03.920ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:367 id = dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool)
2412024-07-25T05:44:04.551ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/8fa4bc78-1026-41f0-b09d-c67224071376): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
2422024-07-25T05:44:04.738ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 1.309974375s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
2432024-07-25T05:44:04.938ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }
2442024-07-25T05:44:05.123ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:332
2452024-07-25T05:44:05.123ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:334
2462024-07-25T05:44:05.123ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:338
2472024-07-25T05:44:05.123ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:340
2482024-07-25T05:44:05.123ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2492024-07-25T05:44:05.123ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2502024-07-25T05:44:05.123ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
2512024-07-25T05:44:05.609ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/8fa4bc78-1026-41f0-b09d-c67224071376): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
2522024-07-25T05:44:05.609ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/8fa4bc78-1026-41f0-b09d-c67224071376): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
2532024-07-25T05:44:05.799ZINFOSledAgent (ServiceManager): Setting up dendrite service file = sled-agent/src/services.rs:2476
2542024-07-25T05:44:05.799ZINFOSledAgent (ServiceManager): Setting up lldpd service file = sled-agent/src/services.rs:2694
2552024-07-25T05:44:05.799ZINFOSledAgent (ServiceManager): Setting up MGS service file = sled-agent/src/services.rs:2346
2562024-07-25T05:44:05.799ZINFOSledAgent (ServiceManager): Setting up wicketd service file = sled-agent/src/services.rs:2393
2572024-07-25T05:44:05.799ZINFOSledAgent (ServiceManager): Setting up mgd service file = sled-agent/src/services.rs:2775
2582024-07-25T05:44:05.799ZINFOSledAgent (ServiceManager): Setting up mg-ddm service file = sled-agent/src/services.rs:2817
2592024-07-25T05:44:05.801ZINFOSledAgent (ServiceManager): Setting up tfport service file = sled-agent/src/services.rs:2619
2602024-07-25T05:44:05.802ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2612024-07-25T05:44:05.998ZINFOSledAgent (ServiceManager): Setting up Simulated SP service file = sled-agent/src/services.rs:2387
2622024-07-25T05:44:05.999ZINFOSledAgent (ServiceManager): Profile for oxz_switch: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/mgs"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12225'/> <propval type="astring" name="id" value='c005a851-a3ef-49f8-8c81-3c9b39f0262d'/> <propval type="astring" name="rack_id" value='b4c6410d-62e1-4369-a8c0-f8e6399dd808'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/wicketd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12226'/> <propval type="astring" name="artifact-address" value='[fdb0:18c0:4d0d:9fb2::2]:12227'/> <propval type="astring" name="baseboard-file" value='/opt/oxide/baseboard.json'/> <propval type="astring" name="mgs-address" value='[::1]:12225'/> <propval type="astring" name="nexus-proxy-address" value='[::]:12229'/> <propval type="astring" name="rack-subnet" value='fd00:1122:3344::'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/switch_zone_setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="gz_local_link_addr" value='fe80::8:20ff:fefc:38b2'/> <property type="astring" name="link_local_links"> <astring_list> <value_node value="tfportrear0_0" /> <value_node value="oxBootstrap0" /> </astring_list> </property> <propval type="astring" name="bootstrap_addr" value='fdb0:18c0:4d0d:9fb2::2'/> <propval type="astring" name="bootstrap_vnic" value='oxBootstrap0'/> <propval type="astring" name="baseboard_info" value='{"type":"pc","identifier":"buskin","model":"i86pc"}'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/dendrite"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_id" value='8fa4bc78-1026-41f0-b09d-c67224071376'/> <propval type="astring" name="rack_id" value='b4c6410d-62e1-4369-a8c0-f8e6399dd808'/> <propval type="astring" name="address" value='[::1]:12224'/> <propval type="astring" name="front_ports" value='1'/> <propval type="astring" name="rear_ports" value='1'/> <propval type="astring" name="port_config" value='/opt/oxide/dendrite/misc/softnpu_single_sled_config.toml'/> <propval type="astring" name="mgmt" value='uds'/> <propval type="astring" name="uds_path" value='/opt/softnpu/stuff'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/tfport"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="host" value='[::1]'/> <propval type="astring" name="port" value='12224'/> <propval type="astring" name="flags" value='--sync-only'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/lldpd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="board_rev" value='softnpu_front_1_rear_1'/> <propval type="astring" name="scrimlet_id" value='buskin'/> <propval type="astring" name="scrimlet_model" value='i86pc'/> <propval type="astring" name="address" value='[::1]:12230'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/pumpkind"> </service> <service version="1" type="service" name="oxide/mgd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_uuid" value='8fa4bc78-1026-41f0-b09d-c67224071376'/> <propval type="astring" name="rack_uuid" value='b4c6410d-62e1-4369-a8c0-f8e6399dd808'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/mg-ddm"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="mode" value='transit'/> <propval type="astring" name="dendrite" value='true'/> <propval type="astring" name="sled_uuid" value='8fa4bc78-1026-41f0-b09d-c67224071376'/> <propval type="astring" name="rack_uuid" value='b4c6410d-62e1-4369-a8c0-f8e6399dd808'/> <propval type="astring" name="interfaces" value='tfportrear0_0/ll'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/uplink"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
2632024-07-25T05:44:05.999ZINFOSledAgent (ServiceManager): Booting oxz_switch zone file = illumos-utils/src/running_zone.rs:465 zone = oxz_switch
2642024-07-25T05:44:06.184ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_dd1c81eb-5ddf-42bb-acfe-348d2d141fca/crypt/zone file = sled-storage/src/dataset.rs:395
2652024-07-25T05:44:06.602ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 2.264922361s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
2662024-07-25T05:44:06.781ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 46.121999ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2672024-07-25T05:44:06.783ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/8fa4bc78-1026-41f0-b09d-c67224071376): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
2682024-07-25T05:44:07.308ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 139.50719ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2692024-07-25T05:44:07.914ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 215.956821ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2702024-07-25T05:44:07.916ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/8fa4bc78-1026-41f0-b09d-c67224071376): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
2712024-07-25T05:44:07.916ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/8fa4bc78-1026-41f0-b09d-c67224071376): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
2722024-07-25T05:44:08.633ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 474.054886ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2732024-07-25T05:44:08.870ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 3.701114401s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
2742024-07-25T05:44:08.879ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_dd1c81eb-5ddf-42bb-acfe-348d2d141fca file = sled-storage/src/dataset.rs:480
2752024-07-25T05:44:09.335ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/8fa4bc78-1026-41f0-b09d-c67224071376): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
2762024-07-25T05:44:09.335ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:512
2772024-07-25T05:44:09.335ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:429
2782024-07-25T05:44:09.335ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:414
2792024-07-25T05:44:09.335ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
2802024-07-25T05:44:09.335ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:127 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } path = /scratch/u2_1.vdev
2812024-07-25T05:44:09.549ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.097160834s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2822024-07-25T05:44:09.559ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_1.vdev file = sled-hardware/src/disk.rs:361
2832024-07-25T05:44:09.559ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:367 id = 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool)
2842024-07-25T05:44:10.007ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/8fa4bc78-1026-41f0-b09d-c67224071376): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
2852024-07-25T05:44:10.965ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 942.720955ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2862024-07-25T05:44:10.965ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/8fa4bc78-1026-41f0-b09d-c67224071376): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
2872024-07-25T05:44:10.975ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }
2882024-07-25T05:44:11.214ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:332
2892024-07-25T05:44:11.214ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:334
2902024-07-25T05:44:11.214ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:338
2912024-07-25T05:44:11.214ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:340
2922024-07-25T05:44:11.214ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2932024-07-25T05:44:11.214ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2942024-07-25T05:44:11.214ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
2952024-07-25T05:44:12.289ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2962024-07-25T05:44:12.795ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/8fa4bc78-1026-41f0-b09d-c67224071376): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
2972024-07-25T05:44:12.796ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/8fa4bc78-1026-41f0-b09d-c67224071376): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
2982024-07-25T05:44:12.796ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 9.144510494s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
2992024-07-25T05:44:12.796ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_59d3cc5a-40b7-44b8-b2af-cfc21755b8bf/crypt/zone file = sled-storage/src/dataset.rs:395
3002024-07-25T05:44:13.070ZINFOSledAgent (ServiceManager): Re-enabling running switch zone (new address) file = sled-agent/src/services.rs:3923 new = [fd00:1122:3344:101::2, ::1] old = [::1]
3012024-07-25T05:44:13.070ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists file = sled-agent/src/services.rs:3939
3022024-07-25T05:44:13.070ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::2, prefix: 64 })) file = illumos-utils/src/running_zone.rs:564 zone = oxz_switch
3032024-07-25T05:44:13.998ZINFOSledAgent (VmmReservoirManager): Finished setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:279 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
3042024-07-25T05:44:14.007ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/8fa4bc78-1026-41f0-b09d-c67224071376): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
3052024-07-25T05:44:14.007ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/8fa4bc78-1026-41f0-b09d-c67224071376): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
3062024-07-25T05:44:14.086ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_59d3cc5a-40b7-44b8-b2af-cfc21755b8bf file = sled-storage/src/dataset.rs:480
3072024-07-25T05:44:14.094ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:512
3082024-07-25T05:44:14.094ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:429
3092024-07-25T05:44:14.094ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:414
3102024-07-25T05:44:14.094ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
3112024-07-25T05:44:14.094ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:127 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } path = /scratch/u2_2.vdev
3122024-07-25T05:44:14.103ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_2.vdev file = sled-hardware/src/disk.rs:361
3132024-07-25T05:44:14.103ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:367 id = e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool)
3142024-07-25T05:44:14.194ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }
3152024-07-25T05:44:14.201ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:332
3162024-07-25T05:44:14.201ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:334
3172024-07-25T05:44:14.201ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:338
3182024-07-25T05:44:14.201ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:340
3192024-07-25T05:44:14.201ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3202024-07-25T05:44:14.201ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3212024-07-25T05:44:14.201ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3222024-07-25T05:44:14.236ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3232024-07-25T05:44:14.243ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_e78b4f40-df20-4bcf-bd8a-b80afa109d70/crypt/zone file = sled-storage/src/dataset.rs:395
3242024-07-25T05:44:14.363ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_e78b4f40-df20-4bcf-bd8a-b80afa109d70 file = sled-storage/src/dataset.rs:480
3252024-07-25T05:44:14.372ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:512
3262024-07-25T05:44:14.372ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:429
3272024-07-25T05:44:14.372ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:414
3282024-07-25T05:44:14.372ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
3292024-07-25T05:44:14.372ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:127 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } path = /scratch/u2_3.vdev
3302024-07-25T05:44:14.382ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_3.vdev file = sled-hardware/src/disk.rs:361
3312024-07-25T05:44:14.382ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:367 id = 63a12260-6721-4f51-85d5-262aedc90628 (zpool)
3322024-07-25T05:44:14.480ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }
3332024-07-25T05:44:14.488ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:332
3342024-07-25T05:44:14.488ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:334
3352024-07-25T05:44:14.488ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:338
3362024-07-25T05:44:14.488ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:340
3372024-07-25T05:44:14.488ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3382024-07-25T05:44:14.488ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3392024-07-25T05:44:14.488ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3402024-07-25T05:44:14.524ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3412024-07-25T05:44:14.532ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_63a12260-6721-4f51-85d5-262aedc90628/crypt/zone file = sled-storage/src/dataset.rs:395
3422024-07-25T05:44:14.658ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_63a12260-6721-4f51-85d5-262aedc90628 file = sled-storage/src/dataset.rs:480
3432024-07-25T05:44:14.666ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:512
3442024-07-25T05:44:14.666ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:429
3452024-07-25T05:44:14.666ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:414
3462024-07-25T05:44:14.666ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
3472024-07-25T05:44:14.666ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:127 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } path = /scratch/u2_4.vdev
3482024-07-25T05:44:14.676ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_4.vdev file = sled-hardware/src/disk.rs:361
3492024-07-25T05:44:14.676ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:367 id = 587239d7-a145-4bb7-b24a-aff014195deb (zpool)
3502024-07-25T05:44:14.769ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }
3512024-07-25T05:44:14.777ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:332
3522024-07-25T05:44:14.777ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:334
3532024-07-25T05:44:14.777ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:338
3542024-07-25T05:44:14.777ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:340
3552024-07-25T05:44:14.777ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3562024-07-25T05:44:14.777ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3572024-07-25T05:44:14.777ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3582024-07-25T05:44:14.813ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3592024-07-25T05:44:14.821ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_587239d7-a145-4bb7-b24a-aff014195deb/crypt/zone file = sled-storage/src/dataset.rs:395
3602024-07-25T05:44:14.943ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_587239d7-a145-4bb7-b24a-aff014195deb file = sled-storage/src/dataset.rs:480
3612024-07-25T05:44:14.952ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:512
3622024-07-25T05:44:14.952ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:429
3632024-07-25T05:44:14.952ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:414
3642024-07-25T05:44:14.952ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
3652024-07-25T05:44:14.952ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:127 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } path = /scratch/u2_5.vdev
3662024-07-25T05:44:14.962ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_5.vdev file = sled-hardware/src/disk.rs:361
3672024-07-25T05:44:14.962ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:367 id = 50e63493-6b9c-42de-846b-797a312b5fbc (zpool)
3682024-07-25T05:44:15.055ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }
3692024-07-25T05:44:15.063ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:332
3702024-07-25T05:44:15.063ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:334
3712024-07-25T05:44:15.063ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:338
3722024-07-25T05:44:15.063ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:340
3732024-07-25T05:44:15.063ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3742024-07-25T05:44:15.063ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3752024-07-25T05:44:15.063ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3762024-07-25T05:44:15.067ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/8fa4bc78-1026-41f0-b09d-c67224071376): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
3772024-07-25T05:44:15.101ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3782024-07-25T05:44:15.109ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_50e63493-6b9c-42de-846b-797a312b5fbc/crypt/zone file = sled-storage/src/dataset.rs:395
3792024-07-25T05:44:15.233ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_50e63493-6b9c-42de-846b-797a312b5fbc file = sled-storage/src/dataset.rs:480
3802024-07-25T05:44:15.241ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:512
3812024-07-25T05:44:15.241ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:429
3822024-07-25T05:44:15.241ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:414
3832024-07-25T05:44:15.241ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
3842024-07-25T05:44:15.241ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:127 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } path = /scratch/u2_6.vdev
3852024-07-25T05:44:15.252ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_6.vdev file = sled-hardware/src/disk.rs:361
3862024-07-25T05:44:15.252ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:367 id = fd75587e-5d72-48df-a44e-8602b2d61469 (zpool)
3872024-07-25T05:44:15.349ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }
3882024-07-25T05:44:15.357ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:332
3892024-07-25T05:44:15.357ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:334
3902024-07-25T05:44:15.357ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:338
3912024-07-25T05:44:15.357ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:340
3922024-07-25T05:44:15.357ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3932024-07-25T05:44:15.357ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3942024-07-25T05:44:15.357ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3952024-07-25T05:44:15.392ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3962024-07-25T05:44:15.400ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_fd75587e-5d72-48df-a44e-8602b2d61469/crypt/zone file = sled-storage/src/dataset.rs:395
3972024-07-25T05:44:15.519ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_fd75587e-5d72-48df-a44e-8602b2d61469 file = sled-storage/src/dataset.rs:480
3982024-07-25T05:44:15.527ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:512
3992024-07-25T05:44:15.527ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:429
4002024-07-25T05:44:15.527ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:414
4012024-07-25T05:44:15.527ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
4022024-07-25T05:44:15.527ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:127 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } path = /scratch/u2_7.vdev
4032024-07-25T05:44:15.536ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_7.vdev file = sled-hardware/src/disk.rs:361
4042024-07-25T05:44:15.536ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:367 id = 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool)
4052024-07-25T05:44:15.626ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }
4062024-07-25T05:44:15.634ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:332
4072024-07-25T05:44:15.634ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:334
4082024-07-25T05:44:15.634ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:338
4092024-07-25T05:44:15.634ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:340
4102024-07-25T05:44:15.634ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4112024-07-25T05:44:15.634ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4122024-07-25T05:44:15.634ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
4132024-07-25T05:44:15.669ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4142024-07-25T05:44:15.677ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_20d995c8-4908-4380-8438-c04a6e3adea2/crypt/zone file = sled-storage/src/dataset.rs:395
4152024-07-25T05:44:15.802ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_20d995c8-4908-4380-8438-c04a6e3adea2 file = sled-storage/src/dataset.rs:480
4162024-07-25T05:44:15.810ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:512
4172024-07-25T05:44:15.810ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:429
4182024-07-25T05:44:15.810ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:414
4192024-07-25T05:44:15.810ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
4202024-07-25T05:44:15.810ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:127 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } path = /scratch/u2_8.vdev
4212024-07-25T05:44:15.819ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_8.vdev file = sled-hardware/src/disk.rs:361
4222024-07-25T05:44:15.819ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:367 id = e43689a1-378e-4be8-9661-3bc522647c6d (zpool)
4232024-07-25T05:44:15.915ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }
4242024-07-25T05:44:15.923ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:332
4252024-07-25T05:44:15.923ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:334
4262024-07-25T05:44:15.923ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:338
4272024-07-25T05:44:15.923ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:340
4282024-07-25T05:44:15.923ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4292024-07-25T05:44:15.923ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4302024-07-25T05:44:15.923ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
4312024-07-25T05:44:15.961ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4322024-07-25T05:44:15.974ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_e43689a1-378e-4be8-9661-3bc522647c6d/crypt/zone file = sled-storage/src/dataset.rs:395
4332024-07-25T05:44:16.001ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/8fa4bc78-1026-41f0-b09d-c67224071376): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
4342024-07-25T05:44:16.070ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/8fa4bc78-1026-41f0-b09d-c67224071376): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
4352024-07-25T05:44:16.105ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_e43689a1-378e-4be8-9661-3bc522647c6d file = sled-storage/src/dataset.rs:480
4362024-07-25T05:44:16.114ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:512
4372024-07-25T05:44:16.114ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:429
4382024-07-25T05:44:16.114ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4392024-07-25T05:44:16.115ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 6d2aa15e-618e-402e-b368-cf27fd8def68 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 75c788e1-45ef-4fcf-958d-db07331bef52 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
4402024-07-25T05:44:16.115ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
4412024-07-25T05:44:16.115ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
4422024-07-25T05:44:16.115ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4432024-07-25T05:44:16.115ZINFOSledAgent: physical disks ensure: Updated storage file = sled-agent/src/sled_agent.rs:828 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
4442024-07-25T05:44:16.115ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
4452024-07-25T05:44:16.115ZINFOSledAgent: physical disks ensure: Propagating new generation of disks file = sled-agent/src/sled_agent.rs:854 generation = Generation(1) sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
4462024-07-25T05:44:16.115ZINFOSledAgent: physical disks ensure: Updated storage monitor file = sled-agent/src/sled_agent.rs:859 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
4472024-07-25T05:44:16.115ZINFOSledAgent: physical disks ensure: Updated zone bundler file = sled-agent/src/sled_agent.rs:864 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
4482024-07-25T05:44:16.115ZINFOSledAgent (ProbeManager): use_only_these_disks: Processing new request file = sled-agent/src/probe_manager.rs:116 gen = Generation(1) sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
4492024-07-25T05:44:16.115ZINFOSledAgent: physical disks ensure: Updated probes file = sled-agent/src/sled_agent.rs:870 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
4502024-07-25T05:44:16.115ZINFOSledAgent (InstanceManager): use_only_these_disks: Processing new request file = sled-agent/src/instance_manager.rs:822 gen = Generation(1)
4512024-07-25T05:44:16.115ZINFOSledAgent: physical disks ensure: Updated instances file = sled-agent/src/sled_agent.rs:875 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
4522024-07-25T05:44:16.115ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 12312412 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:47856 req_id = 95b708ff-48ee-45ce-bc65-f5ab677dafa3 response_code = 200 uri = /omicron-physical-disks
4532024-07-25T05:44:16.146ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(2), zones: [OmicronZoneConfig { id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }] } file = sled-agent/src/rack_setup/service.rs:434 sled_address = [fd00:1122:3344:101::1]:12345
4542024-07-25T05:44:16.146ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42574
4552024-07-25T05:44:16.146ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
4562024-07-25T05:44:16.146ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }
4572024-07-25T05:44:16.198ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
4582024-07-25T05:44:16.198ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }
4592024-07-25T05:44:16.251ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
4602024-07-25T05:44:16.251ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4612024-07-25T05:44:16.303ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
4622024-07-25T05:44:16.303ZINFOSledAgent (ServiceManager): No ledger in /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:188
4632024-07-25T05:44:16.303ZINFOSledAgent (ServiceManager): No ledger in /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:188
4642024-07-25T05:44:16.303ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
4652024-07-25T05:44:16.348ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
4662024-07-25T05:44:16.348ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
4672024-07-25T05:44:16.348ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
4682024-07-25T05:44:16.403ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_0ee9b880-7690-4963-bc6b-3dbf4d39892d file = illumos-utils/src/zone.rs:316
4692024-07-25T05:44:16.407ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_aa38c6ce-70ac-4e2b-a383-8201d1e086e8 file = illumos-utils/src/zone.rs:316
4702024-07-25T05:44:16.411ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_23e793cc-4c0e-4aca-92a8-2b30f6dafa5a file = illumos-utils/src/zone.rs:316
4712024-07-25T05:44:16.423ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_0ee9b880-7690-4963-bc6b-3dbf4d39892d file = illumos-utils/src/zone.rs:354
4722024-07-25T05:44:16.427ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_aa38c6ce-70ac-4e2b-a383-8201d1e086e8 file = illumos-utils/src/zone.rs:354
4732024-07-25T05:44:16.432ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_23e793cc-4c0e-4aca-92a8-2b30f6dafa5a file = illumos-utils/src/zone.rs:354
4742024-07-25T05:44:17.072ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/8fa4bc78-1026-41f0-b09d-c67224071376): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
4752024-07-25T05:44:17.933ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_0ee9b880-7690-4963-bc6b-3dbf4d39892d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService1'/> <propval type="astring" name="gateway" value='fd00:1122:3344:2::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:2::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:2::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:2::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4762024-07-25T05:44:17.933ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:2::, width: 64 }
4772024-07-25T05:44:17.934ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_0ee9b880-7690-4963-bc6b-3dbf4d39892d zone file = illumos-utils/src/running_zone.rs:465 zone = oxz_internal_dns_0ee9b880-7690-4963-bc6b-3dbf4d39892d
4782024-07-25T05:44:18.420ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_aa38c6ce-70ac-4e2b-a383-8201d1e086e8: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService2'/> <propval type="astring" name="gateway" value='fd00:1122:3344:1::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:1::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:1::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:1::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4792024-07-25T05:44:18.898ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_23e793cc-4c0e-4aca-92a8-2b30f6dafa5a: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService3'/> <propval type="astring" name="gateway" value='fd00:1122:3344:3::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:3::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:3::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:3::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4802024-07-25T05:44:18.898ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:1::, width: 64 }
4812024-07-25T05:44:18.898ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:3::, width: 64 }
4822024-07-25T05:44:18.906ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0ee9b880-7690-4963-bc6b-3dbf4d39892d") failed: Property not found. retry in 26.397452ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0ee9b880-7690-4963-bc6b-3dbf4d39892d
4832024-07-25T05:44:18.906ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_aa38c6ce-70ac-4e2b-a383-8201d1e086e8 zone file = illumos-utils/src/running_zone.rs:465 zone = oxz_internal_dns_aa38c6ce-70ac-4e2b-a383-8201d1e086e8
4842024-07-25T05:44:18.910ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_23e793cc-4c0e-4aca-92a8-2b30f6dafa5a zone file = illumos-utils/src/running_zone.rs:465 zone = oxz_internal_dns_23e793cc-4c0e-4aca-92a8-2b30f6dafa5a
4852024-07-25T05:44:18.943ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0ee9b880-7690-4963-bc6b-3dbf4d39892d") failed: Property not found. retry in 143.100954ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0ee9b880-7690-4963-bc6b-3dbf4d39892d
4862024-07-25T05:44:19.040ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_aa38c6ce-70ac-4e2b-a383-8201d1e086e8") failed: Property not found. retry in 46.303751ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_aa38c6ce-70ac-4e2b-a383-8201d1e086e8
4872024-07-25T05:44:19.046ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_23e793cc-4c0e-4aca-92a8-2b30f6dafa5a") failed: Property not found. retry in 29.466021ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_23e793cc-4c0e-4aca-92a8-2b30f6dafa5a
4882024-07-25T05:44:19.082ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_23e793cc-4c0e-4aca-92a8-2b30f6dafa5a") failed: Property not found. retry in 99.926536ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_23e793cc-4c0e-4aca-92a8-2b30f6dafa5a
4892024-07-25T05:44:19.094ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0ee9b880-7690-4963-bc6b-3dbf4d39892d") failed: Property not found. retry in 170.237464ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0ee9b880-7690-4963-bc6b-3dbf4d39892d
4902024-07-25T05:44:19.102ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_aa38c6ce-70ac-4e2b-a383-8201d1e086e8") failed: Property not found. retry in 80.909948ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_aa38c6ce-70ac-4e2b-a383-8201d1e086e8
4912024-07-25T05:44:19.248ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_aa38c6ce-70ac-4e2b-a383-8201d1e086e8") failed: Property not found. retry in 133.583981ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_aa38c6ce-70ac-4e2b-a383-8201d1e086e8
4922024-07-25T05:44:19.314ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_23e793cc-4c0e-4aca-92a8-2b30f6dafa5a") failed: Property not found. retry in 259.948447ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_23e793cc-4c0e-4aca-92a8-2b30f6dafa5a
4932024-07-25T05:44:19.331ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0ee9b880-7690-4963-bc6b-3dbf4d39892d") failed: Property not found. retry in 458.496542ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0ee9b880-7690-4963-bc6b-3dbf4d39892d
4942024-07-25T05:44:19.431ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_aa38c6ce-70ac-4e2b-a383-8201d1e086e8") failed: Property not found. retry in 536.888057ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_aa38c6ce-70ac-4e2b-a383-8201d1e086e8
4952024-07-25T05:44:19.586ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_23e793cc-4c0e-4aca-92a8-2b30f6dafa5a") failed: Property not found. retry in 525.481687ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_23e793cc-4c0e-4aca-92a8-2b30f6dafa5a
4962024-07-25T05:44:19.798ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0ee9b880-7690-4963-bc6b-3dbf4d39892d") failed: Property not found. retry in 1.040577231s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0ee9b880-7690-4963-bc6b-3dbf4d39892d
4972024-07-25T05:44:19.977ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_aa38c6ce-70ac-4e2b-a383-8201d1e086e8") failed: Property not found. retry in 638.40132ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_aa38c6ce-70ac-4e2b-a383-8201d1e086e8
4982024-07-25T05:44:20.123ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_23e793cc-4c0e-4aca-92a8-2b30f6dafa5a") failed: Property not found. retry in 553.274192ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_23e793cc-4c0e-4aca-92a8-2b30f6dafa5a
4992024-07-25T05:44:20.631ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_aa38c6ce-70ac-4e2b-a383-8201d1e086e8") failed: Property not found. retry in 1.173917497s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_aa38c6ce-70ac-4e2b-a383-8201d1e086e8
5002024-07-25T05:44:20.686ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_23e793cc-4c0e-4aca-92a8-2b30f6dafa5a") failed: Property not found. retry in 1.2476454s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_23e793cc-4c0e-4aca-92a8-2b30f6dafa5a
5012024-07-25T05:44:20.848ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0ee9b880-7690-4963-bc6b-3dbf4d39892d") failed: Property not found. retry in 754.060743ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0ee9b880-7690-4963-bc6b-3dbf4d39892d
5022024-07-25T05:44:21.612ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0ee9b880-7690-4963-bc6b-3dbf4d39892d") failed: Property not found. retry in 1.370269374s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0ee9b880-7690-4963-bc6b-3dbf4d39892d
5032024-07-25T05:44:21.813ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_aa38c6ce-70ac-4e2b-a383-8201d1e086e8") failed: Property not found. retry in 514.98304ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_aa38c6ce-70ac-4e2b-a383-8201d1e086e8
5042024-07-25T05:44:21.942ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_23e793cc-4c0e-4aca-92a8-2b30f6dafa5a") failed: Property not found. retry in 1.288095497s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_23e793cc-4c0e-4aca-92a8-2b30f6dafa5a
5052024-07-25T05:44:22.337ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_aa38c6ce-70ac-4e2b-a383-8201d1e086e8") failed: Property not found. retry in 1.191238046s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_aa38c6ce-70ac-4e2b-a383-8201d1e086e8
5062024-07-25T05:44:22.488ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists - OK file = sled-agent/src/services.rs:3947
5072024-07-25T05:44:22.488ZINFOSledAgent (ServiceManager): Ensuring there is a default route file = sled-agent/src/services.rs:3957 gateway = fd00:1122:3344:101::1
5082024-07-25T05:44:22.493ZINFOSledAgent (ServiceManager): configuring dendrite service file = sled-agent/src/services.rs:4036
5092024-07-25T05:44:22.623ZINFOSledAgent (ServiceManager): refreshed dendrite service with new configuration file = sled-agent/src/services.rs:4084
5102024-07-25T05:44:22.623ZINFOSledAgent (ServiceManager): configuring lldp service file = sled-agent/src/services.rs:4111
5112024-07-25T05:44:22.662ZINFOSledAgent (ServiceManager): refreshed lldpd service with new configuration file = sled-agent/src/services.rs:4124
5122024-07-25T05:44:22.662ZINFOSledAgent (ServiceManager): configuring MGS service file = sled-agent/src/services.rs:3988
5132024-07-25T05:44:22.715ZINFOSledAgent (ServiceManager): refreshed MGS service with new configuration file = sled-agent/src/services.rs:4030
5142024-07-25T05:44:22.715ZINFOSledAgent (ServiceManager): configuring wicketd file = sled-agent/src/services.rs:4091 rack_subnet = fd00:1122:3344::
5152024-07-25T05:44:22.736ZINFOSledAgent (ServiceManager): refreshed wicketd service with new configuration file = sled-agent/src/services.rs:4102
5162024-07-25T05:44:22.736ZINFOSledAgent (ServiceManager): configuring mgd service file = sled-agent/src/services.rs:4144
5172024-07-25T05:44:22.829ZINFOSledAgent (ServiceManager): refreshed mgd service with new configuration file = sled-agent/src/services.rs:4176
5182024-07-25T05:44:22.829ZINFOSledAgent (ServiceManager): configuring mg-ddm service file = sled-agent/src/services.rs:4182
5192024-07-25T05:44:22.947ZINFOSledAgent (ServiceManager): refreshed mg-ddm service with new configuration file = sled-agent/src/services.rs:4223
5202024-07-25T05:44:22.947ZINFOSledAgent (ServiceManager): Determining physical location of our switch zone at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:364
5212024-07-25T05:44:22.979ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 38.179016ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/bootstrap/early_networking.rs:383
5222024-07-25T05:44:22.992ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_0ee9b880-7690-4963-bc6b-3dbf4d39892d") failed: Property not found. retry in 531.090446ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_0ee9b880-7690-4963-bc6b-3dbf4d39892d
5232024-07-25T05:44:23.019ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 143.80073ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/bootstrap/early_networking.rs:383
5242024-07-25T05:44:23.165ZINFOSledAgent (ServiceManager): Initializing 1 Uplinks on Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:412
5252024-07-25T05:44:23.196ZINFOSledAgent (ServiceManager): Building Port Configuration file = sled-agent/src/bootstrap/early_networking.rs:651
5262024-07-25T05:44:23.196ZINFOSledAgent (ServiceManager): Checking dendrite uptime file = sled-agent/src/bootstrap/early_networking.rs:689
5272024-07-25T05:44:23.197ZINFOSledAgent (ServiceManager): Dendrite online file = sled-agent/src/bootstrap/early_networking.rs:692 uptime = 0
5282024-07-25T05:44:23.197ZINFOSledAgent (ServiceManager): Configuring default uplink on switch config = PortSettings {\n links: {\n "0": LinkSettings {\n addrs: [\n 10.151.1.174,\n ],\n params: LinkCreate {\n autoneg: false,\n fec: None,\n kr: false,\n lane: Some(\n LinkId(\n 0,\n ),\n ),\n speed: Speed40G,\n },\n },\n },\n} file = sled-agent/src/bootstrap/early_networking.rs:434
5292024-07-25T05:44:23.242ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_23e793cc-4c0e-4aca-92a8-2b30f6dafa5a") failed: Property not found. retry in 1.0106047s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_23e793cc-4c0e-4aca-92a8-2b30f6dafa5a
5302024-07-25T05:44:23.242ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
5312024-07-25T05:44:23.264ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
5322024-07-25T05:44:23.284ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:404
5332024-07-25T05:44:23.284ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5342024-07-25T05:44:23.284ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5352024-07-25T05:44:23.284ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5362024-07-25T05:44:23.284ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5372024-07-25T05:44:23.284ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5382024-07-25T05:44:23.284ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5392024-07-25T05:44:23.284ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5402024-07-25T05:44:23.284ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5412024-07-25T05:44:23.284ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5422024-07-25T05:44:23.284ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5432024-07-25T05:44:23.284ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5442024-07-25T05:44:23.284ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5452024-07-25T05:44:23.284ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5462024-07-25T05:44:23.284ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5472024-07-25T05:44:23.284ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5482024-07-25T05:44:23.284ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5492024-07-25T05:44:23.284ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5502024-07-25T05:44:23.284ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5512024-07-25T05:44:23.284ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5522024-07-25T05:44:23.284ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5532024-07-25T05:44:23.284ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5542024-07-25T05:44:23.284ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5552024-07-25T05:44:23.545ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_aa38c6ce-70ac-4e2b-a383-8201d1e086e8") failed: Property not found. retry in 1.370220737s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_aa38c6ce-70ac-4e2b-a383-8201d1e086e8
5562024-07-25T05:44:24.935ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3062 zone = oxz_internal_dns_0ee9b880-7690-4963-bc6b-3dbf4d39892d
5572024-07-25T05:44:24.935ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3062 zone = oxz_internal_dns_aa38c6ce-70ac-4e2b-a383-8201d1e086e8
5582024-07-25T05:44:24.935ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3062 zone = oxz_internal_dns_23e793cc-4c0e-4aca-92a8-2b30f6dafa5a
5592024-07-25T05:44:24.935ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5602024-07-25T05:44:24.935ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5612024-07-25T05:44:24.935ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 8789225 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:42574 req_id = bcce3176-f156-4ea8-b45b-753af54b39fa response_code = 204 uri = /omicron-zones
5622024-07-25T05:44:24.937ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:1::1]:5353 file = sled-agent/src/rack_setup/service.rs:552
5632024-07-25T05:44:24.982ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:2::1]:5353 file = sled-agent/src/rack_setup/service.rs:552
5642024-07-25T05:44:25.029ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:3::1]:5353 file = sled-agent/src/rack_setup/service.rs:552
5652024-07-25T05:44:25.073ZINFOSledAgent (RSS): Configured all DNS servers file = sled-agent/src/rack_setup/service.rs:588
5662024-07-25T05:44:25.073ZINFOSledAgent (RSS): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:258
5672024-07-25T05:44:25.103ZINFOSledAgent (RSS): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:275
5682024-07-25T05:44:25.104ZINFOSledAgent (RSS): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:300
5692024-07-25T05:44:25.104ZINFOSledAgent (RSS): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:192
5702024-07-25T05:44:25.132ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(3), zones: [OmicronZoneConfig { id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6389ba5c-9d50-4fe4-b1a2-8be3c6861cb0, kind: Service { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c }, name: Name("ntp-3f311cd5-ee38-4707-ab8a-9331491f8e7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 235])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }] } file = sled-agent/src/rack_setup/service.rs:434 sled_address = [fd00:1122:3344:101::1]:12345
5712024-07-25T05:44:25.132ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:59412
5722024-07-25T05:44:25.132ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
5732024-07-25T05:44:25.132ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }
5742024-07-25T05:44:25.152ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
5752024-07-25T05:44:25.152ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }
5762024-07-25T05:44:25.170ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
5772024-07-25T05:44:25.170ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5782024-07-25T05:44:25.188ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
5792024-07-25T05:44:25.188ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:177
5802024-07-25T05:44:25.188ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:177
5812024-07-25T05:44:25.188ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
5822024-07-25T05:44:25.201ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:258
5832024-07-25T05:44:25.233ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:275
5842024-07-25T05:44:25.234ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:300
5852024-07-25T05:44:25.234ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:192
5862024-07-25T05:44:25.263ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:426 port = Port { inner: PortInner(PortData { name: "opte0", ip: 172.30.3.5, mac: MacAddr6([168, 64, 37, 255, 246, 235]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.3.1 } }) }
5872024-07-25T05:44:25.263ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1207 zone_type = ntp
5882024-07-25T05:44:25.264ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:83
5892024-07-25T05:44:25.264ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:90
5902024-07-25T05:44:25.264ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:118
5912024-07-25T05:44:25.264ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
5922024-07-25T05:44:25.288ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_ntp_3f311cd5-ee38-4707-ab8a-9331491f8e7c file = illumos-utils/src/zone.rs:316
5932024-07-25T05:44:25.306ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_ntp_3f311cd5-ee38-4707-ab8a-9331491f8e7c file = illumos-utils/src/zone.rs:354
5942024-07-25T05:44:25.917ZINFOSledAgent (ServiceManager): Profile for oxz_ntp_3f311cd5-ee38-4707-ab8a-9331491f8e7c: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService4'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/chrony-setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="allow" value='fd00:1122:3344:100::/56'/> <propval type="boolean" name="boundary" value='true'/> <propval type="astring" name="server" value='0.pool.ntp.org'/> </property_group> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="1.1.1.1" /> <value_node value="9.9.9.9" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/ntp"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte0'/> <propval type="astring" name="gateway" value='172.30.3.1'/> <propval type="astring" name="ip" value='172.30.3.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
5952024-07-25T05:44:25.917ZINFOSledAgent (ServiceManager): Booting oxz_ntp_3f311cd5-ee38-4707-ab8a-9331491f8e7c zone file = illumos-utils/src/running_zone.rs:465 zone = oxz_ntp_3f311cd5-ee38-4707-ab8a-9331491f8e7c
5962024-07-25T05:44:26.051ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_3f311cd5-ee38-4707-ab8a-9331491f8e7c") failed: Property not found. retry in 51.086364ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_3f311cd5-ee38-4707-ab8a-9331491f8e7c
5972024-07-25T05:44:26.111ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_3f311cd5-ee38-4707-ab8a-9331491f8e7c") failed: Property not found. retry in 134.138304ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_3f311cd5-ee38-4707-ab8a-9331491f8e7c
5982024-07-25T05:44:26.288ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_3f311cd5-ee38-4707-ab8a-9331491f8e7c") failed: Property not found. retry in 232.663274ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_3f311cd5-ee38-4707-ab8a-9331491f8e7c
5992024-07-25T05:44:26.530ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_3f311cd5-ee38-4707-ab8a-9331491f8e7c") failed: Property not found. retry in 444.971337ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_3f311cd5-ee38-4707-ab8a-9331491f8e7c
6002024-07-25T05:44:26.987ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_3f311cd5-ee38-4707-ab8a-9331491f8e7c") failed: Property not found. retry in 504.289589ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_3f311cd5-ee38-4707-ab8a-9331491f8e7c
6012024-07-25T05:44:27.500ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_3f311cd5-ee38-4707-ab8a-9331491f8e7c") failed: Property not found. retry in 930.850343ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_3f311cd5-ee38-4707-ab8a-9331491f8e7c
6022024-07-25T05:44:28.440ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_3f311cd5-ee38-4707-ab8a-9331491f8e7c") failed: Property not found. retry in 1.116323721s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_3f311cd5-ee38-4707-ab8a-9331491f8e7c
6032024-07-25T05:44:29.567ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_3f311cd5-ee38-4707-ab8a-9331491f8e7c") failed: Property not found. retry in 625.92626ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_3f311cd5-ee38-4707-ab8a-9331491f8e7c
6042024-07-25T05:44:30.202ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_3f311cd5-ee38-4707-ab8a-9331491f8e7c") failed: Property not found. retry in 1.435089965s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_3f311cd5-ee38-4707-ab8a-9331491f8e7c
6052024-07-25T05:44:31.656ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3062 zone = oxz_ntp_3f311cd5-ee38-4707-ab8a-9331491f8e7c
6062024-07-25T05:44:31.656ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6072024-07-25T05:44:31.656ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6082024-07-25T05:44:31.656ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 6523934 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59412 req_id = b49e5d71-dd16-41d5-aa18-e48da7660345 response_code = 204 uri = /omicron-zones
6092024-07-25T05:44:31.658ZINFOSledAgent (RSS): Waiting for rack time synchronization file = sled-agent/src/rack_setup/service.rs:629
6102024-07-25T05:44:31.688ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:609
6112024-07-25T05:44:31.688ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:61618
6122024-07-25T05:44:31.697ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_3f311cd5-ee38-4707-ab8a-9331491f8e7c': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3651
6132024-07-25T05:44:31.697ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 9051 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:61618 req_id = 280a9a62-36e7-4a2e-9b2f-f7f94178266c response_code = 500 uri = /timesync
6142024-07-25T05:44:31.700ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:659
6152024-07-25T05:44:31.834ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:609
6162024-07-25T05:44:31.835ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:32781
6172024-07-25T05:44:35.843ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_3f311cd5-ee38-4707-ab8a-9331491f8e7c': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3651
6182024-07-25T05:44:35.843ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 4007957 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:32781 req_id = ec20e060-bffc-4081-b2dd-f18ed7baf527 response_code = 500 uri = /timesync
6192024-07-25T05:44:35.847ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:659
6202024-07-25T05:44:35.986ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:609
6212024-07-25T05:44:35.987ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:61377
6222024-07-25T05:44:35.995ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_3f311cd5-ee38-4707-ab8a-9331491f8e7c': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3651
6232024-07-25T05:44:35.995ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 8522 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:61377 req_id = c58f9cc1-da75-4c16-b432-c03f8418085d response_code = 500 uri = /timesync
6242024-07-25T05:44:35.997ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:659
6252024-07-25T05:44:36.155ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:609
6262024-07-25T05:44:36.155ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:53099
6272024-07-25T05:44:36.163ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 7850 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:53099 req_id = 22f9368b-0178-4e51-b531-b29fe47ec928 response_code = 200 uri = /timesync
6282024-07-25T05:44:36.165ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:637
6292024-07-25T05:44:36.165ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:659
6302024-07-25T05:44:36.312ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:609
6312024-07-25T05:44:36.313ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:49518
6322024-07-25T05:44:36.319ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 6708 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:49518 req_id = faa31035-3896-4766-9752-dee84fc39350 response_code = 200 uri = /timesync
6332024-07-25T05:44:36.321ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:637
6342024-07-25T05:44:36.321ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:659
6352024-07-25T05:44:36.655ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:609
6362024-07-25T05:44:36.655ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:33858
6372024-07-25T05:44:36.662ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 7168 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:33858 req_id = 5c5585e9-50c1-4f0d-84ac-682342d143fc response_code = 200 uri = /timesync
6382024-07-25T05:44:36.664ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:637
6392024-07-25T05:44:36.664ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:659
6402024-07-25T05:44:36.836ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:609
6412024-07-25T05:44:36.836ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:44098
6422024-07-25T05:44:36.843ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 6747 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:44098 req_id = bb08cafb-3ca4-4d51-9b39-13d2c724e61c response_code = 200 uri = /timesync
6432024-07-25T05:44:36.845ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:637
6442024-07-25T05:44:36.845ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:659
6452024-07-25T05:44:37.049ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:609
6462024-07-25T05:44:37.049ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:38676
6472024-07-25T05:44:37.057ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 7235 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:38676 req_id = fb06a834-7719-46db-80ed-4929d9180c1e response_code = 200 uri = /timesync
6482024-07-25T05:44:37.059ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:637
6492024-07-25T05:44:37.059ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:659
6502024-07-25T05:44:37.300ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:609
6512024-07-25T05:44:37.300ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:50734
6522024-07-25T05:44:37.308ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 8131 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:50734 req_id = 564be9a2-38c9-4289-a6bd-0870346b00f1 response_code = 200 uri = /timesync
6532024-07-25T05:44:37.310ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:637
6542024-07-25T05:44:37.310ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:659
6552024-07-25T05:44:37.010ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:609
6562024-07-25T05:44:37.010ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:44710
6572024-07-25T05:44:37.030ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 19537 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:44710 req_id = be8b6594-e6a8-4076-8ec7-202a6d892048 response_code = 200 uri = /timesync
6582024-07-25T05:44:37.032ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: true, ref_id: 2728380539, ip_addr: 162.159.200.123, stratum: 4, ref_time: 1721886277.170336, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:637
6592024-07-25T05:44:37.032ZINFOSledAgent (RSS): Finished setting up Internal DNS and NTP file = sled-agent/src/rack_setup/service.rs:1227
6602024-07-25T05:44:37.062ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6389ba5c-9d50-4fe4-b1a2-8be3c6861cb0, kind: Service { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c }, name: Name("ntp-3f311cd5-ee38-4707-ab8a-9331491f8e7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 235])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 4f6a29ee-ca23-4fa8-be69-497060121028, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:434 sled_address = [fd00:1122:3344:101::1]:12345
6612024-07-25T05:44:37.062ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:33707
6622024-07-25T05:44:37.062ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
6632024-07-25T05:44:37.062ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }
6642024-07-25T05:44:37.081ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
6652024-07-25T05:44:37.081ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }
6662024-07-25T05:44:37.098ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
6672024-07-25T05:44:37.098ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6682024-07-25T05:44:37.117ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
6692024-07-25T05:44:37.117ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6702024-07-25T05:44:37.169ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
6712024-07-25T05:44:37.169ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6722024-07-25T05:44:37.219ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
6732024-07-25T05:44:37.219ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6742024-07-25T05:44:37.269ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
6752024-07-25T05:44:37.269ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6762024-07-25T05:44:37.323ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
6772024-07-25T05:44:37.323ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6782024-07-25T05:44:37.375ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
6792024-07-25T05:44:37.375ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:177
6802024-07-25T05:44:37.375ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:177
6812024-07-25T05:44:37.376ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
6822024-07-25T05:44:37.457ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
6832024-07-25T05:44:37.457ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
6842024-07-25T05:44:37.457ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
6852024-07-25T05:44:37.457ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
6862024-07-25T05:44:37.457ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
6872024-07-25T05:44:37.545ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_4f6a29ee-ca23-4fa8-be69-497060121028 file = illumos-utils/src/zone.rs:316
6882024-07-25T05:44:37.549ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8 file = illumos-utils/src/zone.rs:316
6892024-07-25T05:44:37.554ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_d835fd87-0881-47dd-8e5b-ebb01fd7b0ce file = illumos-utils/src/zone.rs:316
6902024-07-25T05:44:37.558ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_4553d8e1-a7ca-4d20-b1a4-394640a4ad4f file = illumos-utils/src/zone.rs:316
6912024-07-25T05:44:37.562ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_33444b97-3e72-47f7-a319-cbd1ca92b6c8 file = illumos-utils/src/zone.rs:316
6922024-07-25T05:44:37.567ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_4f6a29ee-ca23-4fa8-be69-497060121028 file = illumos-utils/src/zone.rs:354
6932024-07-25T05:44:37.571ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8 file = illumos-utils/src/zone.rs:354
6942024-07-25T05:44:37.576ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_d835fd87-0881-47dd-8e5b-ebb01fd7b0ce file = illumos-utils/src/zone.rs:354
6952024-07-25T05:44:37.580ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_4553d8e1-a7ca-4d20-b1a4-394640a4ad4f file = illumos-utils/src/zone.rs:354
6962024-07-25T05:44:37.585ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_33444b97-3e72-47f7-a319-cbd1ca92b6c8 file = illumos-utils/src/zone.rs:354
6972024-07-25T05:44:40.181ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_4f6a29ee-ca23-4fa8-be69-497060121028: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService5'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::3'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='4f6a29ee-ca23-4fa8-be69-497060121028'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::3]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6982024-07-25T05:44:40.181ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_4f6a29ee-ca23-4fa8-be69-497060121028 zone file = illumos-utils/src/running_zone.rs:465 zone = oxz_cockroachdb_4f6a29ee-ca23-4fa8-be69-497060121028
6992024-07-25T05:44:40.211ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_d835fd87-0881-47dd-8e5b-ebb01fd7b0ce: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService7'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='d835fd87-0881-47dd-8e5b-ebb01fd7b0ce'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::5]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7002024-07-25T05:44:40.211ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_d835fd87-0881-47dd-8e5b-ebb01fd7b0ce zone file = illumos-utils/src/running_zone.rs:465 zone = oxz_cockroachdb_d835fd87-0881-47dd-8e5b-ebb01fd7b0ce
7012024-07-25T05:44:40.217ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService6'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::4'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='3b0743b7-2816-4de6-a40a-d863e46d23f8'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::4]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7022024-07-25T05:44:40.217ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8 zone file = illumos-utils/src/running_zone.rs:465 zone = oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8
7032024-07-25T05:44:40.224ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_4553d8e1-a7ca-4d20-b1a4-394640a4ad4f: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService8'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='4553d8e1-a7ca-4d20-b1a4-394640a4ad4f'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::7]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7042024-07-25T05:44:40.225ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_4553d8e1-a7ca-4d20-b1a4-394640a4ad4f zone file = illumos-utils/src/running_zone.rs:465 zone = oxz_cockroachdb_4553d8e1-a7ca-4d20-b1a4-394640a4ad4f
7052024-07-25T05:44:40.316ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_33444b97-3e72-47f7-a319-cbd1ca92b6c8: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService9'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='33444b97-3e72-47f7-a319-cbd1ca92b6c8'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::6]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7062024-07-25T05:44:40.316ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_33444b97-3e72-47f7-a319-cbd1ca92b6c8 zone file = illumos-utils/src/running_zone.rs:465 zone = oxz_cockroachdb_33444b97-3e72-47f7-a319-cbd1ca92b6c8
7072024-07-25T05:44:40.329ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4f6a29ee-ca23-4fa8-be69-497060121028") failed: Property not found. retry in 57.779835ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4f6a29ee-ca23-4fa8-be69-497060121028
7082024-07-25T05:44:40.351ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d835fd87-0881-47dd-8e5b-ebb01fd7b0ce") failed: Property not found. retry in 26.577257ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d835fd87-0881-47dd-8e5b-ebb01fd7b0ce
7092024-07-25T05:44:40.359ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8") failed: Property not found. retry in 26.907565ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8
7102024-07-25T05:44:40.365ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4553d8e1-a7ca-4d20-b1a4-394640a4ad4f") failed: Property not found. retry in 65.642048ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4553d8e1-a7ca-4d20-b1a4-394640a4ad4f
7112024-07-25T05:44:40.386ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d835fd87-0881-47dd-8e5b-ebb01fd7b0ce") failed: Property not found. retry in 82.981268ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d835fd87-0881-47dd-8e5b-ebb01fd7b0ce
7122024-07-25T05:44:40.407ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8") failed: Property not found. retry in 56.175315ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8
7132024-07-25T05:44:40.442ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4f6a29ee-ca23-4fa8-be69-497060121028") failed: Property not found. retry in 53.594397ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4f6a29ee-ca23-4fa8-be69-497060121028
7142024-07-25T05:44:40.595ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4553d8e1-a7ca-4d20-b1a4-394640a4ad4f") failed: Property not found. retry in 58.188569ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4553d8e1-a7ca-4d20-b1a4-394640a4ad4f
7152024-07-25T05:44:41.095ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 218.440209ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
7162024-07-25T05:44:41.382ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4f6a29ee-ca23-4fa8-be69-497060121028") failed: Property not found. retry in 228.160614ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4f6a29ee-ca23-4fa8-be69-497060121028
7172024-07-25T05:44:41.432ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8") failed: Property not found. retry in 214.745196ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8
7182024-07-25T05:44:41.609ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d835fd87-0881-47dd-8e5b-ebb01fd7b0ce") failed: Property not found. retry in 238.331772ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d835fd87-0881-47dd-8e5b-ebb01fd7b0ce
7192024-07-25T05:44:41.719ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4553d8e1-a7ca-4d20-b1a4-394640a4ad4f") failed: Property not found. retry in 218.315032ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4553d8e1-a7ca-4d20-b1a4-394640a4ad4f
7202024-07-25T05:44:42.086ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_33444b97-3e72-47f7-a319-cbd1ca92b6c8") failed: Property not found. retry in 61.433144ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_33444b97-3e72-47f7-a319-cbd1ca92b6c8
7212024-07-25T05:44:42.307ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4f6a29ee-ca23-4fa8-be69-497060121028") failed: Property not found. retry in 303.237755ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4f6a29ee-ca23-4fa8-be69-497060121028
7222024-07-25T05:44:42.390ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8") failed: Property not found. retry in 521.947244ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8
7232024-07-25T05:44:42.486ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d835fd87-0881-47dd-8e5b-ebb01fd7b0ce") failed: Property not found. retry in 452.137631ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d835fd87-0881-47dd-8e5b-ebb01fd7b0ce
7242024-07-25T05:44:42.556ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4553d8e1-a7ca-4d20-b1a4-394640a4ad4f") failed: Property not found. retry in 222.932796ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4553d8e1-a7ca-4d20-b1a4-394640a4ad4f
7252024-07-25T05:44:42.588ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_33444b97-3e72-47f7-a319-cbd1ca92b6c8") failed: Property not found. retry in 68.159842ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_33444b97-3e72-47f7-a319-cbd1ca92b6c8
7262024-07-25T05:44:42.629ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4f6a29ee-ca23-4fa8-be69-497060121028") failed: Property not found. retry in 413.316286ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4f6a29ee-ca23-4fa8-be69-497060121028
7272024-07-25T05:44:42.676ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_33444b97-3e72-47f7-a319-cbd1ca92b6c8") failed: Property not found. retry in 173.878027ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_33444b97-3e72-47f7-a319-cbd1ca92b6c8
7282024-07-25T05:44:42.801ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4553d8e1-a7ca-4d20-b1a4-394640a4ad4f") failed: Property not found. retry in 993.084116ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4553d8e1-a7ca-4d20-b1a4-394640a4ad4f
7292024-07-25T05:44:42.863ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_33444b97-3e72-47f7-a319-cbd1ca92b6c8") failed: Property not found. retry in 570.249322ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_33444b97-3e72-47f7-a319-cbd1ca92b6c8
7302024-07-25T05:44:42.924ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8") failed: Property not found. retry in 474.450711ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8
7312024-07-25T05:44:42.951ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d835fd87-0881-47dd-8e5b-ebb01fd7b0ce") failed: Property not found. retry in 840.976815ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d835fd87-0881-47dd-8e5b-ebb01fd7b0ce
7322024-07-25T05:44:43.056ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4f6a29ee-ca23-4fa8-be69-497060121028") failed: Property not found. retry in 935.132107ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4f6a29ee-ca23-4fa8-be69-497060121028
7332024-07-25T05:44:43.412ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8") failed: Property not found. retry in 1.255322408s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8
7342024-07-25T05:44:43.447ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_33444b97-3e72-47f7-a319-cbd1ca92b6c8") failed: Property not found. retry in 413.254558ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_33444b97-3e72-47f7-a319-cbd1ca92b6c8
7352024-07-25T05:44:43.804ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d835fd87-0881-47dd-8e5b-ebb01fd7b0ce") failed: Property not found. retry in 1.142773802s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d835fd87-0881-47dd-8e5b-ebb01fd7b0ce
7362024-07-25T05:44:43.812ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4553d8e1-a7ca-4d20-b1a4-394640a4ad4f") failed: Property not found. retry in 713.47099ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4553d8e1-a7ca-4d20-b1a4-394640a4ad4f
7372024-07-25T05:44:43.869ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_33444b97-3e72-47f7-a319-cbd1ca92b6c8") failed: Property not found. retry in 854.592819ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_33444b97-3e72-47f7-a319-cbd1ca92b6c8
7382024-07-25T05:44:44.004ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4f6a29ee-ca23-4fa8-be69-497060121028") failed: Property not found. retry in 867.538176ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4f6a29ee-ca23-4fa8-be69-497060121028
7392024-07-25T05:44:44.537ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4553d8e1-a7ca-4d20-b1a4-394640a4ad4f") failed: Property not found. retry in 751.17122ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4553d8e1-a7ca-4d20-b1a4-394640a4ad4f
7402024-07-25T05:44:44.680ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8") failed: Property not found. retry in 544.783176ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8
7412024-07-25T05:44:44.735ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_33444b97-3e72-47f7-a319-cbd1ca92b6c8") failed: Property not found. retry in 811.377988ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_33444b97-3e72-47f7-a319-cbd1ca92b6c8
7422024-07-25T05:44:44.882ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4f6a29ee-ca23-4fa8-be69-497060121028") failed: Property not found. retry in 1.371962857s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4f6a29ee-ca23-4fa8-be69-497060121028
7432024-07-25T05:44:44.959ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d835fd87-0881-47dd-8e5b-ebb01fd7b0ce") failed: Property not found. retry in 660.375253ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d835fd87-0881-47dd-8e5b-ebb01fd7b0ce
7442024-07-25T05:44:45.237ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8") failed: Property not found. retry in 754.880337ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8
7452024-07-25T05:44:45.302ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4553d8e1-a7ca-4d20-b1a4-394640a4ad4f") failed: Property not found. retry in 669.450852ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4553d8e1-a7ca-4d20-b1a4-394640a4ad4f
7462024-07-25T05:44:45.559ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_33444b97-3e72-47f7-a319-cbd1ca92b6c8") failed: Property not found. retry in 1.382036708s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_33444b97-3e72-47f7-a319-cbd1ca92b6c8
7472024-07-25T05:44:45.632ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d835fd87-0881-47dd-8e5b-ebb01fd7b0ce") failed: Property not found. retry in 822.488873ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d835fd87-0881-47dd-8e5b-ebb01fd7b0ce
7482024-07-25T05:44:45.983ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4553d8e1-a7ca-4d20-b1a4-394640a4ad4f") failed: Property not found. retry in 955.606974ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4553d8e1-a7ca-4d20-b1a4-394640a4ad4f
7492024-07-25T05:44:46.002ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8") failed: Property not found. retry in 974.520096ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8
7502024-07-25T05:44:46.268ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4f6a29ee-ca23-4fa8-be69-497060121028") failed: Property not found. retry in 1.327877443s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4f6a29ee-ca23-4fa8-be69-497060121028
7512024-07-25T05:44:46.467ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d835fd87-0881-47dd-8e5b-ebb01fd7b0ce") failed: Property not found. retry in 1.069057515s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d835fd87-0881-47dd-8e5b-ebb01fd7b0ce
7522024-07-25T05:44:46.952ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4553d8e1-a7ca-4d20-b1a4-394640a4ad4f") failed: Property not found. retry in 869.566296ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4553d8e1-a7ca-4d20-b1a4-394640a4ad4f
7532024-07-25T05:44:46.961ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_33444b97-3e72-47f7-a319-cbd1ca92b6c8") failed: Property not found. retry in 643.688368ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_33444b97-3e72-47f7-a319-cbd1ca92b6c8
7542024-07-25T05:44:46.989ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8") failed: Property not found. retry in 596.785626ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8
7552024-07-25T05:44:47.551ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d835fd87-0881-47dd-8e5b-ebb01fd7b0ce") failed: Property not found. retry in 679.679674ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d835fd87-0881-47dd-8e5b-ebb01fd7b0ce
7562024-07-25T05:44:47.597ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8") failed: Property not found. retry in 1.076627973s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8
7572024-07-25T05:44:47.608ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4f6a29ee-ca23-4fa8-be69-497060121028") failed: Property not found. retry in 675.2868ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4f6a29ee-ca23-4fa8-be69-497060121028
7582024-07-25T05:44:47.616ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_33444b97-3e72-47f7-a319-cbd1ca92b6c8") failed: Property not found. retry in 1.012982768s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_33444b97-3e72-47f7-a319-cbd1ca92b6c8
7592024-07-25T05:44:47.833ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4553d8e1-a7ca-4d20-b1a4-394640a4ad4f") failed: Property not found. retry in 733.732424ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4553d8e1-a7ca-4d20-b1a4-394640a4ad4f
7602024-07-25T05:44:48.242ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d835fd87-0881-47dd-8e5b-ebb01fd7b0ce") failed: Property not found. retry in 834.821527ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d835fd87-0881-47dd-8e5b-ebb01fd7b0ce
7612024-07-25T05:44:48.297ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4f6a29ee-ca23-4fa8-be69-497060121028") failed: Property not found. retry in 1.351120523s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4f6a29ee-ca23-4fa8-be69-497060121028
7622024-07-25T05:44:48.578ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4553d8e1-a7ca-4d20-b1a4-394640a4ad4f") failed: Property not found. retry in 1.236057011s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4553d8e1-a7ca-4d20-b1a4-394640a4ad4f
7632024-07-25T05:44:48.641ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_33444b97-3e72-47f7-a319-cbd1ca92b6c8") failed: Property not found. retry in 522.717903ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_33444b97-3e72-47f7-a319-cbd1ca92b6c8
7642024-07-25T05:44:48.685ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8") failed: Property not found. retry in 896.52462ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8
7652024-07-25T05:44:49.088ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d835fd87-0881-47dd-8e5b-ebb01fd7b0ce") failed: Property not found. retry in 600.910805ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d835fd87-0881-47dd-8e5b-ebb01fd7b0ce
7662024-07-25T05:44:49.174ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_33444b97-3e72-47f7-a319-cbd1ca92b6c8") failed: Property not found. retry in 1.19663353s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_33444b97-3e72-47f7-a319-cbd1ca92b6c8
7672024-07-25T05:44:49.594ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8") failed: Property not found. retry in 754.480489ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8
7682024-07-25T05:44:49.662ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4f6a29ee-ca23-4fa8-be69-497060121028") failed: Property not found. retry in 778.640707ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4f6a29ee-ca23-4fa8-be69-497060121028
7692024-07-25T05:44:49.700ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d835fd87-0881-47dd-8e5b-ebb01fd7b0ce") failed: Property not found. retry in 1.331698231s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d835fd87-0881-47dd-8e5b-ebb01fd7b0ce
7702024-07-25T05:44:49.826ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4553d8e1-a7ca-4d20-b1a4-394640a4ad4f") failed: Property not found. retry in 658.537264ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4553d8e1-a7ca-4d20-b1a4-394640a4ad4f
7712024-07-25T05:44:50.362ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8") failed: Property not found. retry in 1.290330039s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8
7722024-07-25T05:44:50.383ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_33444b97-3e72-47f7-a319-cbd1ca92b6c8") failed: Property not found. retry in 1.496309738s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_33444b97-3e72-47f7-a319-cbd1ca92b6c8
7732024-07-25T05:44:50.451ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4f6a29ee-ca23-4fa8-be69-497060121028") failed: Property not found. retry in 1.438157084s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4f6a29ee-ca23-4fa8-be69-497060121028
7742024-07-25T05:44:51.046ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d835fd87-0881-47dd-8e5b-ebb01fd7b0ce") failed: Property not found. retry in 1.305752233s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d835fd87-0881-47dd-8e5b-ebb01fd7b0ce
7752024-07-25T05:44:51.666ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8") failed: Property not found. retry in 680.740592ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8
7762024-07-25T05:44:51.815ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:49883
7772024-07-25T05:44:51.815ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 88 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:49883 req_id = 58b7e820-3a36-40e7-ab67-fbfe0ca61613 response_code = 200 uri = /baseboard
7782024-07-25T05:44:52.358ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8") failed: Property not found. retry in 889.753104ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8
7792024-07-25T05:44:53.259ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8") failed: Property not found. retry in 1.307726611s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8
7802024-07-25T05:44:54.580ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8") failed: Property not found. retry in 1.07919368s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8
7812024-07-25T05:44:55.687ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8") failed: Property not found. retry in 817.559347ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8
7822024-07-25T05:44:56.517ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8") failed: Property not found. retry in 1.451834132s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8
7832024-07-25T05:44:56.697ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 488.064022ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
7842024-07-25T05:44:57.984ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8") failed: Property not found. retry in 1.214057976s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8
7852024-07-25T05:44:59.209ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8") failed: Property not found. retry in 798.180637ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8
7862024-07-25T05:45:00.020ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8") failed: Property not found. retry in 1.038391511s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8
7872024-07-25T05:45:01.072ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8") failed: Property not found. retry in 625.174019ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8
7882024-07-25T05:45:01.720ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_4f6a29ee-ca23-4fa8-be69-497060121028
7892024-07-25T05:45:01.720ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_3b0743b7-2816-4de6-a40a-d863e46d23f8
7902024-07-25T05:45:01.720ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_d835fd87-0881-47dd-8e5b-ebb01fd7b0ce
7912024-07-25T05:45:01.720ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_4553d8e1-a7ca-4d20-b1a4-394640a4ad4f
7922024-07-25T05:45:01.720ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_33444b97-3e72-47f7-a319-cbd1ca92b6c8
7932024-07-25T05:45:01.720ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
7942024-07-25T05:45:01.725ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
7952024-07-25T05:45:01.735ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 24901126 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33707 req_id = accc0bbb-e5fe-4345-a34b-277637f391e4 response_code = 204 uri = /omicron-zones
7962024-07-25T05:45:01.767ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:63729
7972024-07-25T05:45:01.795ZINFOSledAgent (ServiceManager): Initializing CRDB Cluster - sending request to [fd00:1122:3344:101::6]:32221 file = sled-agent/src/services.rs:3485
7982024-07-25T05:45:02.554ZINFOSledAgent (ServiceManager): Formatting CRDB file = sled-agent/src/services.rs:3503
7992024-07-25T05:45:20.833ZINFOSledAgent (ServiceManager): Formatting CRDB - Completed file = sled-agent/src/services.rs:3526
8002024-07-25T05:45:20.833ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 19015150 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:63729 req_id = 5164ed42-f50a-435a-8244-1b5ec385180f response_code = 204 uri = /cockroachdb
8012024-07-25T05:45:20.838ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 925.847787ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
8022024-07-25T05:45:20.868ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6389ba5c-9d50-4fe4-b1a2-8be3c6861cb0, kind: Service { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c }, name: Name("ntp-3f311cd5-ee38-4707-ab8a-9331491f8e7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 235])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 4f6a29ee-ca23-4fa8-be69-497060121028, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: b04db43b-dc24-43ce-9d97-32193f88e046, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 2c3f8453-233f-428f-93cc-0bfefb84929d, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: e4efbc31-e513-4290-82f5-c51d8afea0c7, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 221c5e7c-0832-4890-8b0a-f9ba08725876, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4b4998f4-10de-49f2-9326-b482137c5fd3, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: a635cc62-7516-4fd9-b490-62d6de386095, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ac9fa851-e7e1-4295-ac79-d6076526f465, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 1cb96976-2047-44b8-a79c-9fcd74d17bfa, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: a0b9fba0-c330-402f-9bc0-a7510786c98a, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d94d9602-d610-44ce-b9fc-06ded41a1480, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb1f626-0f55-4385-87de-de59052d98d2, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External } } } }, OmicronZoneConfig { id: adff3617-5320-4e42-8b10-1430a1eb0afc, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:434 sled_address = [fd00:1122:3344:101::1]:12345
8032024-07-25T05:45:20.868ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:47495
8042024-07-25T05:45:20.869ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
8052024-07-25T05:45:20.869ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }
8062024-07-25T05:45:20.890ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
8072024-07-25T05:45:20.890ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }
8082024-07-25T05:45:20.912ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
8092024-07-25T05:45:20.912ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8102024-07-25T05:45:20.932ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
8112024-07-25T05:45:20.932ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8122024-07-25T05:45:20.953ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
8132024-07-25T05:45:20.953ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8142024-07-25T05:45:20.972ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
8152024-07-25T05:45:20.972ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8162024-07-25T05:45:20.992ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
8172024-07-25T05:45:20.992ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8182024-07-25T05:45:21.012ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
8192024-07-25T05:45:21.012ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8202024-07-25T05:45:21.035ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
8212024-07-25T05:45:21.035ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }
8222024-07-25T05:45:21.155ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
8232024-07-25T05:45:21.156ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }
8242024-07-25T05:45:21.257ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
8252024-07-25T05:45:21.257ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }
8262024-07-25T05:45:21.380ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
8272024-07-25T05:45:21.380ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }
8282024-07-25T05:45:21.485ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
8292024-07-25T05:45:21.485ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }
8302024-07-25T05:45:21.600ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
8312024-07-25T05:45:21.600ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }
8322024-07-25T05:45:21.717ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
8332024-07-25T05:45:21.717ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }
8342024-07-25T05:45:21.870ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
8352024-07-25T05:45:21.870ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }
8362024-07-25T05:45:21.931ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:37914
8372024-07-25T05:45:21.931ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 59 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:37914 req_id = 97226c11-239c-4ed3-8ee5-62de403a8c5b response_code = 200 uri = /baseboard
8382024-07-25T05:45:22.013ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
8392024-07-25T05:45:22.013ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }
8402024-07-25T05:45:22.134ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
8412024-07-25T05:45:22.134ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }
8422024-07-25T05:45:22.257ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
8432024-07-25T05:45:22.257ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }
8442024-07-25T05:45:22.354ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
8452024-07-25T05:45:22.354ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }
8462024-07-25T05:45:22.451ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
8472024-07-25T05:45:22.452ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:177
8482024-07-25T05:45:22.452ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:177
8492024-07-25T05:45:22.452ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
8502024-07-25T05:45:22.552ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
8512024-07-25T05:45:22.552ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
8522024-07-25T05:45:22.552ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
8532024-07-25T05:45:22.552ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
8542024-07-25T05:45:22.552ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
8552024-07-25T05:45:22.552ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
8562024-07-25T05:45:22.552ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
8572024-07-25T05:45:22.552ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
8582024-07-25T05:45:22.552ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
8592024-07-25T05:45:22.552ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
8602024-07-25T05:45:22.552ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
8612024-07-25T05:45:22.552ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
8622024-07-25T05:45:22.552ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
8632024-07-25T05:45:22.552ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
8642024-07-25T05:45:22.552ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
8652024-07-25T05:45:22.552ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
8662024-07-25T05:45:22.552ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
8672024-07-25T05:45:22.552ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
8682024-07-25T05:45:22.552ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
8692024-07-25T05:45:22.552ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2", TimeNotSynchronized), ("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9", TimeNotSynchronized), ("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480", TimeNotSynchronized), ("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046", TimeNotSynchronized), ("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e", TimeNotSynchronized), ("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b", TimeNotSynchronized), ("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa", TimeNotSynchronized), ("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc", TimeNotSynchronized), ("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7", TimeNotSynchronized), ("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c", TimeNotSynchronized), ("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465", TimeNotSynchronized), ("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79", TimeNotSynchronized), ("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d", TimeNotSynchronized), ("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876", TimeNotSynchronized), ("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862", TimeNotSynchronized), ("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3", TimeNotSynchronized), ("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095", TimeNotSynchronized), ("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a", TimeNotSynchronized), ("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 1679288 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:47495 req_id = 691a1dfe-1f82-42d7-8d71-6e986e50417f response_code = 500 uri = /omicron-zones
8702024-07-25T05:45:22.553ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "691a1dfe-1f82-42d7-8d71-6e986e50417f", "content-length": "124", "date": "Thu, 25 Jul 2024 05:45:21 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "691a1dfe-1f82-42d7-8d71-6e986e50417f" } file = sled-agent/src/rack_setup/service.rs:472 retry_after = 107.192945ms sled_address = [fd00:1122:3344:101::1]:12345
8712024-07-25T05:45:22.661ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6389ba5c-9d50-4fe4-b1a2-8be3c6861cb0, kind: Service { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c }, name: Name("ntp-3f311cd5-ee38-4707-ab8a-9331491f8e7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 235])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 4f6a29ee-ca23-4fa8-be69-497060121028, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: b04db43b-dc24-43ce-9d97-32193f88e046, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 2c3f8453-233f-428f-93cc-0bfefb84929d, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: e4efbc31-e513-4290-82f5-c51d8afea0c7, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 221c5e7c-0832-4890-8b0a-f9ba08725876, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4b4998f4-10de-49f2-9326-b482137c5fd3, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: a635cc62-7516-4fd9-b490-62d6de386095, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ac9fa851-e7e1-4295-ac79-d6076526f465, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 1cb96976-2047-44b8-a79c-9fcd74d17bfa, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: a0b9fba0-c330-402f-9bc0-a7510786c98a, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d94d9602-d610-44ce-b9fc-06ded41a1480, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb1f626-0f55-4385-87de-de59052d98d2, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External } } } }, OmicronZoneConfig { id: adff3617-5320-4e42-8b10-1430a1eb0afc, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:434 sled_address = [fd00:1122:3344:101::1]:12345
8722024-07-25T05:45:22.662ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
8732024-07-25T05:45:22.662ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }
8742024-07-25T05:45:22.684ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
8752024-07-25T05:45:22.684ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }
8762024-07-25T05:45:22.704ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
8772024-07-25T05:45:22.704ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8782024-07-25T05:45:22.724ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
8792024-07-25T05:45:22.724ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8802024-07-25T05:45:22.742ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
8812024-07-25T05:45:22.742ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8822024-07-25T05:45:22.760ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
8832024-07-25T05:45:22.760ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8842024-07-25T05:45:22.780ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
8852024-07-25T05:45:22.781ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8862024-07-25T05:45:22.801ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
8872024-07-25T05:45:22.801ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8882024-07-25T05:45:22.821ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
8892024-07-25T05:45:22.821ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }
8902024-07-25T05:45:22.840ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
8912024-07-25T05:45:22.840ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }
8922024-07-25T05:45:22.860ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
8932024-07-25T05:45:22.860ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }
8942024-07-25T05:45:22.878ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
8952024-07-25T05:45:22.878ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }
8962024-07-25T05:45:22.900ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
8972024-07-25T05:45:22.900ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }
8982024-07-25T05:45:22.922ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
8992024-07-25T05:45:22.922ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }
9002024-07-25T05:45:22.942ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
9012024-07-25T05:45:22.942ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }
9022024-07-25T05:45:22.963ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
9032024-07-25T05:45:22.963ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }
9042024-07-25T05:45:22.982ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
9052024-07-25T05:45:22.982ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }
9062024-07-25T05:45:23.002ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
9072024-07-25T05:45:23.002ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }
9082024-07-25T05:45:23.022ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
9092024-07-25T05:45:23.022ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }
9102024-07-25T05:45:23.042ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
9112024-07-25T05:45:23.042ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }
9122024-07-25T05:45:23.061ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
9132024-07-25T05:45:23.061ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:177
9142024-07-25T05:45:23.061ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:177
9152024-07-25T05:45:23.061ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
9162024-07-25T05:45:23.156ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
9172024-07-25T05:45:23.156ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
9182024-07-25T05:45:23.156ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
9192024-07-25T05:45:23.156ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
9202024-07-25T05:45:23.156ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
9212024-07-25T05:45:23.156ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
9222024-07-25T05:45:23.156ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
9232024-07-25T05:45:23.156ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
9242024-07-25T05:45:23.156ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
9252024-07-25T05:45:23.156ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
9262024-07-25T05:45:23.156ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
9272024-07-25T05:45:23.156ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
9282024-07-25T05:45:23.156ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
9292024-07-25T05:45:23.156ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
9302024-07-25T05:45:23.156ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
9312024-07-25T05:45:23.156ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
9322024-07-25T05:45:23.156ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
9332024-07-25T05:45:23.156ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
9342024-07-25T05:45:23.156ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
9352024-07-25T05:45:23.156ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79", TimeNotSynchronized), ("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a", TimeNotSynchronized), ("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa", TimeNotSynchronized), ("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc", TimeNotSynchronized), ("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e", TimeNotSynchronized), ("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c", TimeNotSynchronized), ("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862", TimeNotSynchronized), ("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7", TimeNotSynchronized), ("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9", TimeNotSynchronized), ("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465", TimeNotSynchronized), ("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876", TimeNotSynchronized), ("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3", TimeNotSynchronized), ("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c", TimeNotSynchronized), ("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046", TimeNotSynchronized), ("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d", TimeNotSynchronized), ("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b", TimeNotSynchronized), ("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095", TimeNotSynchronized), ("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480", TimeNotSynchronized), ("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 493429 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:47495 req_id = c25a7f8f-6a51-48a0-a5a1-63a7a430a1cc response_code = 500 uri = /omicron-zones
9362024-07-25T05:45:23.157ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "c25a7f8f-6a51-48a0-a5a1-63a7a430a1cc", "content-length": "124", "date": "Thu, 25 Jul 2024 05:45:23 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "c25a7f8f-6a51-48a0-a5a1-63a7a430a1cc" } file = sled-agent/src/rack_setup/service.rs:472 retry_after = 76.267732ms sled_address = [fd00:1122:3344:101::1]:12345
9372024-07-25T05:45:23.235ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6389ba5c-9d50-4fe4-b1a2-8be3c6861cb0, kind: Service { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c }, name: Name("ntp-3f311cd5-ee38-4707-ab8a-9331491f8e7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 235])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 4f6a29ee-ca23-4fa8-be69-497060121028, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: b04db43b-dc24-43ce-9d97-32193f88e046, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 2c3f8453-233f-428f-93cc-0bfefb84929d, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: e4efbc31-e513-4290-82f5-c51d8afea0c7, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 221c5e7c-0832-4890-8b0a-f9ba08725876, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4b4998f4-10de-49f2-9326-b482137c5fd3, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: a635cc62-7516-4fd9-b490-62d6de386095, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ac9fa851-e7e1-4295-ac79-d6076526f465, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 1cb96976-2047-44b8-a79c-9fcd74d17bfa, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: a0b9fba0-c330-402f-9bc0-a7510786c98a, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d94d9602-d610-44ce-b9fc-06ded41a1480, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb1f626-0f55-4385-87de-de59052d98d2, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External } } } }, OmicronZoneConfig { id: adff3617-5320-4e42-8b10-1430a1eb0afc, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:434 sled_address = [fd00:1122:3344:101::1]:12345
9382024-07-25T05:45:23.235ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
9392024-07-25T05:45:23.235ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }
9402024-07-25T05:45:23.258ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
9412024-07-25T05:45:23.258ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }
9422024-07-25T05:45:23.279ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
9432024-07-25T05:45:23.279ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9442024-07-25T05:45:23.298ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
9452024-07-25T05:45:23.298ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9462024-07-25T05:45:23.319ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
9472024-07-25T05:45:23.319ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9482024-07-25T05:45:23.338ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
9492024-07-25T05:45:23.338ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9502024-07-25T05:45:23.356ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
9512024-07-25T05:45:23.356ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9522024-07-25T05:45:23.375ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
9532024-07-25T05:45:23.375ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9542024-07-25T05:45:23.394ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
9552024-07-25T05:45:23.394ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }
9562024-07-25T05:45:23.413ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
9572024-07-25T05:45:23.413ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }
9582024-07-25T05:45:23.432ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
9592024-07-25T05:45:23.432ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }
9602024-07-25T05:45:23.451ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
9612024-07-25T05:45:23.451ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }
9622024-07-25T05:45:23.470ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
9632024-07-25T05:45:23.470ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }
9642024-07-25T05:45:23.492ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
9652024-07-25T05:45:23.492ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }
9662024-07-25T05:45:23.515ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
9672024-07-25T05:45:23.515ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }
9682024-07-25T05:45:23.536ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
9692024-07-25T05:45:23.536ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }
9702024-07-25T05:45:23.556ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
9712024-07-25T05:45:23.556ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }
9722024-07-25T05:45:23.577ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
9732024-07-25T05:45:23.577ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }
9742024-07-25T05:45:23.597ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
9752024-07-25T05:45:23.597ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }
9762024-07-25T05:45:23.617ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
9772024-07-25T05:45:23.617ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }
9782024-07-25T05:45:23.637ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
9792024-07-25T05:45:23.637ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:177
9802024-07-25T05:45:23.637ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:177
9812024-07-25T05:45:23.637ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
9822024-07-25T05:45:23.733ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
9832024-07-25T05:45:23.733ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
9842024-07-25T05:45:23.733ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
9852024-07-25T05:45:23.733ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
9862024-07-25T05:45:23.733ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
9872024-07-25T05:45:23.733ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
9882024-07-25T05:45:23.733ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
9892024-07-25T05:45:23.733ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
9902024-07-25T05:45:23.733ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
9912024-07-25T05:45:23.733ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
9922024-07-25T05:45:23.733ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
9932024-07-25T05:45:23.733ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
9942024-07-25T05:45:23.733ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
9952024-07-25T05:45:23.733ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
9962024-07-25T05:45:23.733ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
9972024-07-25T05:45:23.733ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
9982024-07-25T05:45:23.733ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
9992024-07-25T05:45:23.733ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
10002024-07-25T05:45:23.733ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
10012024-07-25T05:45:23.733ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046", TimeNotSynchronized), ("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2", TimeNotSynchronized), ("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862", TimeNotSynchronized), ("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc", TimeNotSynchronized), ("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d", TimeNotSynchronized), ("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3", TimeNotSynchronized), ("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a", TimeNotSynchronized), ("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c", TimeNotSynchronized), ("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b", TimeNotSynchronized), ("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa", TimeNotSynchronized), ("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480", TimeNotSynchronized), ("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095", TimeNotSynchronized), ("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79", TimeNotSynchronized), ("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7", TimeNotSynchronized), ("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465", TimeNotSynchronized), ("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c", TimeNotSynchronized), ("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876", TimeNotSynchronized), ("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e", TimeNotSynchronized), ("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 496421 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:47495 req_id = dea6787a-2fba-4c76-ac82-dc1c0043f9ec response_code = 500 uri = /omicron-zones
10022024-07-25T05:45:23.733ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "dea6787a-2fba-4c76-ac82-dc1c0043f9ec", "content-length": "124", "date": "Thu, 25 Jul 2024 05:45:23 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "dea6787a-2fba-4c76-ac82-dc1c0043f9ec" } file = sled-agent/src/rack_setup/service.rs:472 retry_after = 115.119741ms sled_address = [fd00:1122:3344:101::1]:12345
10032024-07-25T05:45:23.849ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6389ba5c-9d50-4fe4-b1a2-8be3c6861cb0, kind: Service { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c }, name: Name("ntp-3f311cd5-ee38-4707-ab8a-9331491f8e7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 235])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 4f6a29ee-ca23-4fa8-be69-497060121028, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: b04db43b-dc24-43ce-9d97-32193f88e046, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 2c3f8453-233f-428f-93cc-0bfefb84929d, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: e4efbc31-e513-4290-82f5-c51d8afea0c7, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 221c5e7c-0832-4890-8b0a-f9ba08725876, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4b4998f4-10de-49f2-9326-b482137c5fd3, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: a635cc62-7516-4fd9-b490-62d6de386095, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ac9fa851-e7e1-4295-ac79-d6076526f465, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 1cb96976-2047-44b8-a79c-9fcd74d17bfa, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: a0b9fba0-c330-402f-9bc0-a7510786c98a, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d94d9602-d610-44ce-b9fc-06ded41a1480, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb1f626-0f55-4385-87de-de59052d98d2, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External } } } }, OmicronZoneConfig { id: adff3617-5320-4e42-8b10-1430a1eb0afc, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:434 sled_address = [fd00:1122:3344:101::1]:12345
10042024-07-25T05:45:23.849ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
10052024-07-25T05:45:23.849ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }
10062024-07-25T05:45:23.872ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
10072024-07-25T05:45:23.872ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }
10082024-07-25T05:45:23.894ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
10092024-07-25T05:45:23.894ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10102024-07-25T05:45:23.916ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
10112024-07-25T05:45:23.916ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10122024-07-25T05:45:23.938ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
10132024-07-25T05:45:23.938ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10142024-07-25T05:45:23.957ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
10152024-07-25T05:45:23.958ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10162024-07-25T05:45:23.976ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
10172024-07-25T05:45:23.976ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10182024-07-25T05:45:23.995ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
10192024-07-25T05:45:23.995ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10202024-07-25T05:45:24.015ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
10212024-07-25T05:45:24.015ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }
10222024-07-25T05:45:24.033ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
10232024-07-25T05:45:24.033ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }
10242024-07-25T05:45:24.053ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
10252024-07-25T05:45:24.053ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }
10262024-07-25T05:45:24.073ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
10272024-07-25T05:45:24.073ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }
10282024-07-25T05:45:24.092ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
10292024-07-25T05:45:24.092ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }
10302024-07-25T05:45:24.111ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
10312024-07-25T05:45:24.111ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }
10322024-07-25T05:45:24.131ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
10332024-07-25T05:45:24.132ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }
10342024-07-25T05:45:24.150ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
10352024-07-25T05:45:24.150ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }
10362024-07-25T05:45:24.168ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
10372024-07-25T05:45:24.168ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }
10382024-07-25T05:45:24.186ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
10392024-07-25T05:45:24.186ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }
10402024-07-25T05:45:24.204ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
10412024-07-25T05:45:24.204ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }
10422024-07-25T05:45:24.223ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
10432024-07-25T05:45:24.223ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }
10442024-07-25T05:45:24.241ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
10452024-07-25T05:45:24.241ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:177
10462024-07-25T05:45:24.242ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:177
10472024-07-25T05:45:24.242ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
10482024-07-25T05:45:24.340ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
10492024-07-25T05:45:24.340ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
10502024-07-25T05:45:24.340ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
10512024-07-25T05:45:24.340ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
10522024-07-25T05:45:24.340ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
10532024-07-25T05:45:24.340ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
10542024-07-25T05:45:24.340ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
10552024-07-25T05:45:24.340ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
10562024-07-25T05:45:24.340ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
10572024-07-25T05:45:24.340ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
10582024-07-25T05:45:24.340ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
10592024-07-25T05:45:24.340ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
10602024-07-25T05:45:24.340ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
10612024-07-25T05:45:24.340ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
10622024-07-25T05:45:24.340ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
10632024-07-25T05:45:24.340ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
10642024-07-25T05:45:24.340ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
10652024-07-25T05:45:24.340ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
10662024-07-25T05:45:24.340ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
10672024-07-25T05:45:24.340ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480", TimeNotSynchronized), ("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7", TimeNotSynchronized), ("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095", TimeNotSynchronized), ("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3", TimeNotSynchronized), ("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d", TimeNotSynchronized), ("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876", TimeNotSynchronized), ("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465", TimeNotSynchronized), ("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a", TimeNotSynchronized), ("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c", TimeNotSynchronized), ("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046", TimeNotSynchronized), ("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e", TimeNotSynchronized), ("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c", TimeNotSynchronized), ("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc", TimeNotSynchronized), ("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9", TimeNotSynchronized), ("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862", TimeNotSynchronized), ("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa", TimeNotSynchronized), ("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79", TimeNotSynchronized), ("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2", TimeNotSynchronized), ("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 489672 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:47495 req_id = 4fe74a12-ba05-4cd8-9d0b-9875e61ad626 response_code = 500 uri = /omicron-zones
10682024-07-25T05:45:24.340ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "4fe74a12-ba05-4cd8-9d0b-9875e61ad626", "content-length": "124", "date": "Thu, 25 Jul 2024 05:45:24 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "4fe74a12-ba05-4cd8-9d0b-9875e61ad626" } file = sled-agent/src/rack_setup/service.rs:472 retry_after = 153.645353ms sled_address = [fd00:1122:3344:101::1]:12345
10692024-07-25T05:45:24.495ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6389ba5c-9d50-4fe4-b1a2-8be3c6861cb0, kind: Service { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c }, name: Name("ntp-3f311cd5-ee38-4707-ab8a-9331491f8e7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 235])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 4f6a29ee-ca23-4fa8-be69-497060121028, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: b04db43b-dc24-43ce-9d97-32193f88e046, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 2c3f8453-233f-428f-93cc-0bfefb84929d, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: e4efbc31-e513-4290-82f5-c51d8afea0c7, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 221c5e7c-0832-4890-8b0a-f9ba08725876, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4b4998f4-10de-49f2-9326-b482137c5fd3, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: a635cc62-7516-4fd9-b490-62d6de386095, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ac9fa851-e7e1-4295-ac79-d6076526f465, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 1cb96976-2047-44b8-a79c-9fcd74d17bfa, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: a0b9fba0-c330-402f-9bc0-a7510786c98a, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d94d9602-d610-44ce-b9fc-06ded41a1480, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb1f626-0f55-4385-87de-de59052d98d2, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External } } } }, OmicronZoneConfig { id: adff3617-5320-4e42-8b10-1430a1eb0afc, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:434 sled_address = [fd00:1122:3344:101::1]:12345
10702024-07-25T05:45:24.496ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
10712024-07-25T05:45:24.496ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }
10722024-07-25T05:45:24.519ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
10732024-07-25T05:45:24.519ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }
10742024-07-25T05:45:24.539ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
10752024-07-25T05:45:24.539ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10762024-07-25T05:45:24.558ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
10772024-07-25T05:45:24.558ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10782024-07-25T05:45:24.577ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
10792024-07-25T05:45:24.577ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10802024-07-25T05:45:24.596ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
10812024-07-25T05:45:24.596ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10822024-07-25T05:45:24.616ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
10832024-07-25T05:45:24.616ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10842024-07-25T05:45:24.636ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
10852024-07-25T05:45:24.636ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10862024-07-25T05:45:24.652ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
10872024-07-25T05:45:24.652ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }
10882024-07-25T05:45:24.669ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
10892024-07-25T05:45:24.669ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }
10902024-07-25T05:45:24.688ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
10912024-07-25T05:45:24.688ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }
10922024-07-25T05:45:24.708ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
10932024-07-25T05:45:24.708ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }
10942024-07-25T05:45:24.726ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
10952024-07-25T05:45:24.726ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }
10962024-07-25T05:45:24.746ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
10972024-07-25T05:45:24.746ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }
10982024-07-25T05:45:24.766ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
10992024-07-25T05:45:24.766ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }
11002024-07-25T05:45:24.786ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
11012024-07-25T05:45:24.786ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }
11022024-07-25T05:45:24.805ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
11032024-07-25T05:45:24.805ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }
11042024-07-25T05:45:24.824ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
11052024-07-25T05:45:24.824ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }
11062024-07-25T05:45:24.845ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
11072024-07-25T05:45:24.845ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }
11082024-07-25T05:45:24.863ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
11092024-07-25T05:45:24.863ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }
11102024-07-25T05:45:24.882ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
11112024-07-25T05:45:24.882ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:177
11122024-07-25T05:45:24.882ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:177
11132024-07-25T05:45:24.882ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
11142024-07-25T05:45:24.975ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
11152024-07-25T05:45:24.975ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
11162024-07-25T05:45:24.975ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
11172024-07-25T05:45:24.975ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
11182024-07-25T05:45:24.975ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
11192024-07-25T05:45:24.975ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
11202024-07-25T05:45:24.975ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
11212024-07-25T05:45:24.975ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
11222024-07-25T05:45:24.975ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
11232024-07-25T05:45:24.975ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
11242024-07-25T05:45:24.975ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
11252024-07-25T05:45:24.975ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
11262024-07-25T05:45:24.975ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
11272024-07-25T05:45:24.975ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
11282024-07-25T05:45:24.975ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
11292024-07-25T05:45:24.975ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
11302024-07-25T05:45:24.975ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
11312024-07-25T05:45:24.975ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
11322024-07-25T05:45:24.975ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
11332024-07-25T05:45:24.976ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e", TimeNotSynchronized), ("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2", TimeNotSynchronized), ("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876", TimeNotSynchronized), ("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa", TimeNotSynchronized), ("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095", TimeNotSynchronized), ("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c", TimeNotSynchronized), ("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc", TimeNotSynchronized), ("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9", TimeNotSynchronized), ("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480", TimeNotSynchronized), ("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3", TimeNotSynchronized), ("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465", TimeNotSynchronized), ("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c", TimeNotSynchronized), ("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7", TimeNotSynchronized), ("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d", TimeNotSynchronized), ("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b", TimeNotSynchronized), ("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862", TimeNotSynchronized), ("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a", TimeNotSynchronized), ("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046", TimeNotSynchronized), ("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 478944 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:47495 req_id = dafd868d-d39f-4a0e-88ad-63b01a7b10dc response_code = 500 uri = /omicron-zones
11342024-07-25T05:45:24.976ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "dafd868d-d39f-4a0e-88ad-63b01a7b10dc", "content-length": "124", "date": "Thu, 25 Jul 2024 05:45:24 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "dafd868d-d39f-4a0e-88ad-63b01a7b10dc" } file = sled-agent/src/rack_setup/service.rs:472 retry_after = 111.028637ms sled_address = [fd00:1122:3344:101::1]:12345
11352024-07-25T05:45:25.088ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6389ba5c-9d50-4fe4-b1a2-8be3c6861cb0, kind: Service { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c }, name: Name("ntp-3f311cd5-ee38-4707-ab8a-9331491f8e7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 235])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 4f6a29ee-ca23-4fa8-be69-497060121028, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: b04db43b-dc24-43ce-9d97-32193f88e046, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 2c3f8453-233f-428f-93cc-0bfefb84929d, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: e4efbc31-e513-4290-82f5-c51d8afea0c7, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 221c5e7c-0832-4890-8b0a-f9ba08725876, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4b4998f4-10de-49f2-9326-b482137c5fd3, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: a635cc62-7516-4fd9-b490-62d6de386095, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ac9fa851-e7e1-4295-ac79-d6076526f465, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 1cb96976-2047-44b8-a79c-9fcd74d17bfa, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: a0b9fba0-c330-402f-9bc0-a7510786c98a, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d94d9602-d610-44ce-b9fc-06ded41a1480, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb1f626-0f55-4385-87de-de59052d98d2, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External } } } }, OmicronZoneConfig { id: adff3617-5320-4e42-8b10-1430a1eb0afc, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:434 sled_address = [fd00:1122:3344:101::1]:12345
11362024-07-25T05:45:25.089ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
11372024-07-25T05:45:25.089ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }
11382024-07-25T05:45:25.112ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
11392024-07-25T05:45:25.113ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }
11402024-07-25T05:45:25.134ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
11412024-07-25T05:45:25.134ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11422024-07-25T05:45:25.154ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
11432024-07-25T05:45:25.154ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11442024-07-25T05:45:25.175ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
11452024-07-25T05:45:25.175ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11462024-07-25T05:45:25.193ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
11472024-07-25T05:45:25.193ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11482024-07-25T05:45:25.211ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
11492024-07-25T05:45:25.211ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11502024-07-25T05:45:25.229ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
11512024-07-25T05:45:25.229ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11522024-07-25T05:45:25.252ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
11532024-07-25T05:45:25.252ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }
11542024-07-25T05:45:25.272ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
11552024-07-25T05:45:25.272ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }
11562024-07-25T05:45:25.293ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
11572024-07-25T05:45:25.293ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }
11582024-07-25T05:45:25.312ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
11592024-07-25T05:45:25.312ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }
11602024-07-25T05:45:25.334ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
11612024-07-25T05:45:25.334ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }
11622024-07-25T05:45:25.354ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
11632024-07-25T05:45:25.354ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }
11642024-07-25T05:45:25.375ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
11652024-07-25T05:45:25.376ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }
11662024-07-25T05:45:25.396ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
11672024-07-25T05:45:25.396ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }
11682024-07-25T05:45:25.416ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
11692024-07-25T05:45:25.416ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }
11702024-07-25T05:45:25.436ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
11712024-07-25T05:45:25.436ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }
11722024-07-25T05:45:25.455ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
11732024-07-25T05:45:25.455ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }
11742024-07-25T05:45:25.473ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
11752024-07-25T05:45:25.473ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }
11762024-07-25T05:45:25.494ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
11772024-07-25T05:45:25.494ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:177
11782024-07-25T05:45:25.494ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:177
11792024-07-25T05:45:25.495ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
11802024-07-25T05:45:25.593ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
11812024-07-25T05:45:25.593ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
11822024-07-25T05:45:25.593ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
11832024-07-25T05:45:25.593ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
11842024-07-25T05:45:25.593ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
11852024-07-25T05:45:25.593ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
11862024-07-25T05:45:25.593ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
11872024-07-25T05:45:25.593ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
11882024-07-25T05:45:25.593ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
11892024-07-25T05:45:25.593ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
11902024-07-25T05:45:25.593ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
11912024-07-25T05:45:25.593ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
11922024-07-25T05:45:25.593ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
11932024-07-25T05:45:25.593ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
11942024-07-25T05:45:25.593ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
11952024-07-25T05:45:25.593ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
11962024-07-25T05:45:25.593ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
11972024-07-25T05:45:25.593ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
11982024-07-25T05:45:25.593ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
11992024-07-25T05:45:25.593ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480", TimeNotSynchronized), ("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046", TimeNotSynchronized), ("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9", TimeNotSynchronized), ("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465", TimeNotSynchronized), ("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3", TimeNotSynchronized), ("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa", TimeNotSynchronized), ("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876", TimeNotSynchronized), ("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2", TimeNotSynchronized), ("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b", TimeNotSynchronized), ("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095", TimeNotSynchronized), ("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7", TimeNotSynchronized), ("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d", TimeNotSynchronized), ("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c", TimeNotSynchronized), ("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c", TimeNotSynchronized), ("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc", TimeNotSynchronized), ("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e", TimeNotSynchronized), ("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79", TimeNotSynchronized), ("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862", TimeNotSynchronized), ("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 503451 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:47495 req_id = 9ed53cac-69e7-4694-aba1-678cc90b837e response_code = 500 uri = /omicron-zones
12002024-07-25T05:45:25.593ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "9ed53cac-69e7-4694-aba1-678cc90b837e", "content-length": "124", "date": "Thu, 25 Jul 2024 05:45:24 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "9ed53cac-69e7-4694-aba1-678cc90b837e" } file = sled-agent/src/rack_setup/service.rs:472 retry_after = 188.5872ms sled_address = [fd00:1122:3344:101::1]:12345
12012024-07-25T05:45:25.783ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6389ba5c-9d50-4fe4-b1a2-8be3c6861cb0, kind: Service { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c }, name: Name("ntp-3f311cd5-ee38-4707-ab8a-9331491f8e7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 235])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 4f6a29ee-ca23-4fa8-be69-497060121028, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: b04db43b-dc24-43ce-9d97-32193f88e046, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 2c3f8453-233f-428f-93cc-0bfefb84929d, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: e4efbc31-e513-4290-82f5-c51d8afea0c7, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 221c5e7c-0832-4890-8b0a-f9ba08725876, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4b4998f4-10de-49f2-9326-b482137c5fd3, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: a635cc62-7516-4fd9-b490-62d6de386095, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ac9fa851-e7e1-4295-ac79-d6076526f465, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 1cb96976-2047-44b8-a79c-9fcd74d17bfa, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: a0b9fba0-c330-402f-9bc0-a7510786c98a, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d94d9602-d610-44ce-b9fc-06ded41a1480, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb1f626-0f55-4385-87de-de59052d98d2, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External } } } }, OmicronZoneConfig { id: adff3617-5320-4e42-8b10-1430a1eb0afc, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:434 sled_address = [fd00:1122:3344:101::1]:12345
12022024-07-25T05:45:25.784ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
12032024-07-25T05:45:25.784ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }
12042024-07-25T05:45:25.808ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
12052024-07-25T05:45:25.808ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }
12062024-07-25T05:45:25.830ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
12072024-07-25T05:45:25.830ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12082024-07-25T05:45:25.849ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
12092024-07-25T05:45:25.849ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12102024-07-25T05:45:25.869ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
12112024-07-25T05:45:25.869ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12122024-07-25T05:45:25.892ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
12132024-07-25T05:45:25.892ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12142024-07-25T05:45:25.913ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
12152024-07-25T05:45:25.913ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12162024-07-25T05:45:25.932ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
12172024-07-25T05:45:25.932ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12182024-07-25T05:45:25.951ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
12192024-07-25T05:45:25.951ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }
12202024-07-25T05:45:25.970ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
12212024-07-25T05:45:25.970ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }
12222024-07-25T05:45:25.990ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
12232024-07-25T05:45:25.990ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }
12242024-07-25T05:45:26.010ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
12252024-07-25T05:45:26.010ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }
12262024-07-25T05:45:26.028ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
12272024-07-25T05:45:26.028ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }
12282024-07-25T05:45:26.047ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
12292024-07-25T05:45:26.047ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }
12302024-07-25T05:45:26.067ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
12312024-07-25T05:45:26.067ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }
12322024-07-25T05:45:26.088ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
12332024-07-25T05:45:26.088ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }
12342024-07-25T05:45:26.107ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
12352024-07-25T05:45:26.107ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }
12362024-07-25T05:45:26.126ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
12372024-07-25T05:45:26.126ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }
12382024-07-25T05:45:26.144ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
12392024-07-25T05:45:26.144ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }
12402024-07-25T05:45:26.164ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
12412024-07-25T05:45:26.164ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }
12422024-07-25T05:45:26.184ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
12432024-07-25T05:45:26.184ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:177
12442024-07-25T05:45:26.184ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:177
12452024-07-25T05:45:26.184ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
12462024-07-25T05:45:26.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
12472024-07-25T05:45:26.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
12482024-07-25T05:45:26.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
12492024-07-25T05:45:26.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
12502024-07-25T05:45:26.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
12512024-07-25T05:45:26.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
12522024-07-25T05:45:26.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
12532024-07-25T05:45:26.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
12542024-07-25T05:45:26.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
12552024-07-25T05:45:26.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
12562024-07-25T05:45:26.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
12572024-07-25T05:45:26.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
12582024-07-25T05:45:26.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
12592024-07-25T05:45:26.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
12602024-07-25T05:45:26.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
12612024-07-25T05:45:26.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
12622024-07-25T05:45:26.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
12632024-07-25T05:45:26.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
12642024-07-25T05:45:26.279ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
12652024-07-25T05:45:26.279ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b", TimeNotSynchronized), ("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79", TimeNotSynchronized), ("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465", TimeNotSynchronized), ("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7", TimeNotSynchronized), ("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3", TimeNotSynchronized), ("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d", TimeNotSynchronized), ("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876", TimeNotSynchronized), ("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862", TimeNotSynchronized), ("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa", TimeNotSynchronized), ("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e", TimeNotSynchronized), ("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046", TimeNotSynchronized), ("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480", TimeNotSynchronized), ("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a", TimeNotSynchronized), ("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c", TimeNotSynchronized), ("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c", TimeNotSynchronized), ("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2", TimeNotSynchronized), ("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9", TimeNotSynchronized), ("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc", TimeNotSynchronized), ("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 494199 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:47495 req_id = 4375fe17-7094-4c02-a1be-da1577d7573d response_code = 500 uri = /omicron-zones
12662024-07-25T05:45:26.279ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "4375fe17-7094-4c02-a1be-da1577d7573d", "content-length": "124", "date": "Thu, 25 Jul 2024 05:45:25 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "4375fe17-7094-4c02-a1be-da1577d7573d" } file = sled-agent/src/rack_setup/service.rs:472 retry_after = 270.851555ms sled_address = [fd00:1122:3344:101::1]:12345
12672024-07-25T05:45:26.551ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6389ba5c-9d50-4fe4-b1a2-8be3c6861cb0, kind: Service { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c }, name: Name("ntp-3f311cd5-ee38-4707-ab8a-9331491f8e7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 235])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 4f6a29ee-ca23-4fa8-be69-497060121028, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: b04db43b-dc24-43ce-9d97-32193f88e046, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 2c3f8453-233f-428f-93cc-0bfefb84929d, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: e4efbc31-e513-4290-82f5-c51d8afea0c7, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 221c5e7c-0832-4890-8b0a-f9ba08725876, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4b4998f4-10de-49f2-9326-b482137c5fd3, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: a635cc62-7516-4fd9-b490-62d6de386095, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ac9fa851-e7e1-4295-ac79-d6076526f465, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 1cb96976-2047-44b8-a79c-9fcd74d17bfa, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: a0b9fba0-c330-402f-9bc0-a7510786c98a, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d94d9602-d610-44ce-b9fc-06ded41a1480, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb1f626-0f55-4385-87de-de59052d98d2, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External } } } }, OmicronZoneConfig { id: adff3617-5320-4e42-8b10-1430a1eb0afc, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:434 sled_address = [fd00:1122:3344:101::1]:12345
12682024-07-25T05:45:26.552ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
12692024-07-25T05:45:26.552ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }
12702024-07-25T05:45:26.577ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
12712024-07-25T05:45:26.577ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }
12722024-07-25T05:45:26.598ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
12732024-07-25T05:45:26.598ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12742024-07-25T05:45:26.618ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
12752024-07-25T05:45:26.618ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12762024-07-25T05:45:26.638ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
12772024-07-25T05:45:26.638ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12782024-07-25T05:45:26.658ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
12792024-07-25T05:45:26.658ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12802024-07-25T05:45:26.679ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
12812024-07-25T05:45:26.679ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12822024-07-25T05:45:26.699ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
12832024-07-25T05:45:26.699ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12842024-07-25T05:45:26.718ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
12852024-07-25T05:45:26.718ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }
12862024-07-25T05:45:26.738ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
12872024-07-25T05:45:26.738ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }
12882024-07-25T05:45:26.757ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
12892024-07-25T05:45:26.757ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }
12902024-07-25T05:45:26.777ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
12912024-07-25T05:45:26.777ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }
12922024-07-25T05:45:26.798ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
12932024-07-25T05:45:26.798ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }
12942024-07-25T05:45:26.819ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
12952024-07-25T05:45:26.819ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }
12962024-07-25T05:45:26.839ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
12972024-07-25T05:45:26.839ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }
12982024-07-25T05:45:26.859ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
12992024-07-25T05:45:26.859ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }
13002024-07-25T05:45:26.878ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
13012024-07-25T05:45:26.878ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }
13022024-07-25T05:45:26.898ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
13032024-07-25T05:45:26.898ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }
13042024-07-25T05:45:26.919ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
13052024-07-25T05:45:26.919ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }
13062024-07-25T05:45:26.939ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
13072024-07-25T05:45:26.939ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }
13082024-07-25T05:45:26.959ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
13092024-07-25T05:45:26.959ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:177
13102024-07-25T05:45:26.960ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:177
13112024-07-25T05:45:26.960ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
13122024-07-25T05:45:27.061ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
13132024-07-25T05:45:27.061ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
13142024-07-25T05:45:27.061ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
13152024-07-25T05:45:27.061ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
13162024-07-25T05:45:27.061ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
13172024-07-25T05:45:27.061ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
13182024-07-25T05:45:27.061ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
13192024-07-25T05:45:27.061ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
13202024-07-25T05:45:27.061ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
13212024-07-25T05:45:27.061ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
13222024-07-25T05:45:27.061ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
13232024-07-25T05:45:27.061ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
13242024-07-25T05:45:27.061ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
13252024-07-25T05:45:27.061ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
13262024-07-25T05:45:27.061ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
13272024-07-25T05:45:27.061ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
13282024-07-25T05:45:27.061ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
13292024-07-25T05:45:27.061ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
13302024-07-25T05:45:27.061ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
13312024-07-25T05:45:27.061ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b", TimeNotSynchronized), ("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d", TimeNotSynchronized), ("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876", TimeNotSynchronized), ("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa", TimeNotSynchronized), ("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c", TimeNotSynchronized), ("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e", TimeNotSynchronized), ("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7", TimeNotSynchronized), ("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095", TimeNotSynchronized), ("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79", TimeNotSynchronized), ("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3", TimeNotSynchronized), ("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046", TimeNotSynchronized), ("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465", TimeNotSynchronized), ("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862", TimeNotSynchronized), ("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480", TimeNotSynchronized), ("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c", TimeNotSynchronized), ("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc", TimeNotSynchronized), ("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9", TimeNotSynchronized), ("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a", TimeNotSynchronized), ("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 508497 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:47495 req_id = f9f0da75-d24b-4ff0-b736-44da53249c06 response_code = 500 uri = /omicron-zones
13322024-07-25T05:45:27.062ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "f9f0da75-d24b-4ff0-b736-44da53249c06", "content-length": "124", "date": "Thu, 25 Jul 2024 05:45:26 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "f9f0da75-d24b-4ff0-b736-44da53249c06" } file = sled-agent/src/rack_setup/service.rs:472 retry_after = 279.317867ms sled_address = [fd00:1122:3344:101::1]:12345
13332024-07-25T05:45:27.343ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6389ba5c-9d50-4fe4-b1a2-8be3c6861cb0, kind: Service { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c }, name: Name("ntp-3f311cd5-ee38-4707-ab8a-9331491f8e7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 235])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 4f6a29ee-ca23-4fa8-be69-497060121028, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: b04db43b-dc24-43ce-9d97-32193f88e046, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 2c3f8453-233f-428f-93cc-0bfefb84929d, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: e4efbc31-e513-4290-82f5-c51d8afea0c7, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 221c5e7c-0832-4890-8b0a-f9ba08725876, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4b4998f4-10de-49f2-9326-b482137c5fd3, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: a635cc62-7516-4fd9-b490-62d6de386095, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ac9fa851-e7e1-4295-ac79-d6076526f465, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 1cb96976-2047-44b8-a79c-9fcd74d17bfa, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: a0b9fba0-c330-402f-9bc0-a7510786c98a, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d94d9602-d610-44ce-b9fc-06ded41a1480, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb1f626-0f55-4385-87de-de59052d98d2, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External } } } }, OmicronZoneConfig { id: adff3617-5320-4e42-8b10-1430a1eb0afc, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:434 sled_address = [fd00:1122:3344:101::1]:12345
13342024-07-25T05:45:27.344ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
13352024-07-25T05:45:27.344ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }
13362024-07-25T05:45:27.367ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
13372024-07-25T05:45:27.367ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }
13382024-07-25T05:45:27.387ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
13392024-07-25T05:45:27.387ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13402024-07-25T05:45:27.408ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
13412024-07-25T05:45:27.408ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13422024-07-25T05:45:27.427ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
13432024-07-25T05:45:27.428ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13442024-07-25T05:45:27.448ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
13452024-07-25T05:45:27.448ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13462024-07-25T05:45:27.467ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
13472024-07-25T05:45:27.467ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13482024-07-25T05:45:27.486ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
13492024-07-25T05:45:27.486ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13502024-07-25T05:45:27.504ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
13512024-07-25T05:45:27.504ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }
13522024-07-25T05:45:27.524ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
13532024-07-25T05:45:27.524ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }
13542024-07-25T05:45:27.543ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
13552024-07-25T05:45:27.543ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }
13562024-07-25T05:45:27.564ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
13572024-07-25T05:45:27.564ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }
13582024-07-25T05:45:27.582ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
13592024-07-25T05:45:27.582ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }
13602024-07-25T05:45:27.602ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
13612024-07-25T05:45:27.602ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }
13622024-07-25T05:45:27.620ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
13632024-07-25T05:45:27.621ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }
13642024-07-25T05:45:27.641ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
13652024-07-25T05:45:27.641ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }
13662024-07-25T05:45:27.662ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
13672024-07-25T05:45:27.662ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }
13682024-07-25T05:45:27.681ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
13692024-07-25T05:45:27.681ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }
13702024-07-25T05:45:27.702ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
13712024-07-25T05:45:27.702ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }
13722024-07-25T05:45:27.723ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
13732024-07-25T05:45:27.723ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }
13742024-07-25T05:45:27.743ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
13752024-07-25T05:45:27.743ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:177
13762024-07-25T05:45:27.743ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:177
13772024-07-25T05:45:27.744ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
13782024-07-25T05:45:27.843ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
13792024-07-25T05:45:27.844ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
13802024-07-25T05:45:27.844ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
13812024-07-25T05:45:27.844ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
13822024-07-25T05:45:27.844ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
13832024-07-25T05:45:27.844ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
13842024-07-25T05:45:27.844ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
13852024-07-25T05:45:27.844ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
13862024-07-25T05:45:27.844ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
13872024-07-25T05:45:27.844ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
13882024-07-25T05:45:27.844ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
13892024-07-25T05:45:27.844ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
13902024-07-25T05:45:27.844ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
13912024-07-25T05:45:27.844ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
13922024-07-25T05:45:27.844ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
13932024-07-25T05:45:27.844ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
13942024-07-25T05:45:27.844ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
13952024-07-25T05:45:27.844ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
13962024-07-25T05:45:27.844ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
13972024-07-25T05:45:27.844ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b", TimeNotSynchronized), ("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9", TimeNotSynchronized), ("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095", TimeNotSynchronized), ("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2", TimeNotSynchronized), ("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862", TimeNotSynchronized), ("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e", TimeNotSynchronized), ("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480", TimeNotSynchronized), ("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c", TimeNotSynchronized), ("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79", TimeNotSynchronized), ("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7", TimeNotSynchronized), ("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3", TimeNotSynchronized), ("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d", TimeNotSynchronized), ("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa", TimeNotSynchronized), ("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a", TimeNotSynchronized), ("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c", TimeNotSynchronized), ("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046", TimeNotSynchronized), ("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc", TimeNotSynchronized), ("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465", TimeNotSynchronized), ("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 499554 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:47495 req_id = f9684fbf-a16a-481f-b7e8-91821c9486d8 response_code = 500 uri = /omicron-zones
13982024-07-25T05:45:27.844ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "f9684fbf-a16a-481f-b7e8-91821c9486d8", "content-length": "124", "date": "Thu, 25 Jul 2024 05:45:27 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "f9684fbf-a16a-481f-b7e8-91821c9486d8" } file = sled-agent/src/rack_setup/service.rs:472 retry_after = 325.924107ms sled_address = [fd00:1122:3344:101::1]:12345
13992024-07-25T05:45:28.172ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6389ba5c-9d50-4fe4-b1a2-8be3c6861cb0, kind: Service { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c }, name: Name("ntp-3f311cd5-ee38-4707-ab8a-9331491f8e7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 235])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 4f6a29ee-ca23-4fa8-be69-497060121028, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: b04db43b-dc24-43ce-9d97-32193f88e046, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 2c3f8453-233f-428f-93cc-0bfefb84929d, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: e4efbc31-e513-4290-82f5-c51d8afea0c7, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 221c5e7c-0832-4890-8b0a-f9ba08725876, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4b4998f4-10de-49f2-9326-b482137c5fd3, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: a635cc62-7516-4fd9-b490-62d6de386095, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ac9fa851-e7e1-4295-ac79-d6076526f465, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 1cb96976-2047-44b8-a79c-9fcd74d17bfa, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: a0b9fba0-c330-402f-9bc0-a7510786c98a, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d94d9602-d610-44ce-b9fc-06ded41a1480, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb1f626-0f55-4385-87de-de59052d98d2, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External } } } }, OmicronZoneConfig { id: adff3617-5320-4e42-8b10-1430a1eb0afc, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:434 sled_address = [fd00:1122:3344:101::1]:12345
14002024-07-25T05:45:28.173ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
14012024-07-25T05:45:28.173ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }
14022024-07-25T05:45:28.197ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
14032024-07-25T05:45:28.197ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }
14042024-07-25T05:45:28.218ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
14052024-07-25T05:45:28.218ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14062024-07-25T05:45:28.239ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
14072024-07-25T05:45:28.239ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14082024-07-25T05:45:28.258ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
14092024-07-25T05:45:28.258ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14102024-07-25T05:45:28.278ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
14112024-07-25T05:45:28.278ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14122024-07-25T05:45:28.296ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
14132024-07-25T05:45:28.296ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14142024-07-25T05:45:28.315ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
14152024-07-25T05:45:28.315ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14162024-07-25T05:45:28.335ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
14172024-07-25T05:45:28.335ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }
14182024-07-25T05:45:28.356ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
14192024-07-25T05:45:28.356ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }
14202024-07-25T05:45:28.376ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
14212024-07-25T05:45:28.376ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }
14222024-07-25T05:45:28.398ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
14232024-07-25T05:45:28.398ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }
14242024-07-25T05:45:28.417ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
14252024-07-25T05:45:28.417ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }
14262024-07-25T05:45:28.436ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
14272024-07-25T05:45:28.436ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }
14282024-07-25T05:45:28.455ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
14292024-07-25T05:45:28.455ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }
14302024-07-25T05:45:28.475ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
14312024-07-25T05:45:28.476ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }
14322024-07-25T05:45:28.496ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
14332024-07-25T05:45:28.496ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }
14342024-07-25T05:45:28.518ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
14352024-07-25T05:45:28.518ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }
14362024-07-25T05:45:28.539ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
14372024-07-25T05:45:28.539ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }
14382024-07-25T05:45:28.560ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
14392024-07-25T05:45:28.560ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }
14402024-07-25T05:45:28.582ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
14412024-07-25T05:45:28.582ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:177
14422024-07-25T05:45:28.582ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:177
14432024-07-25T05:45:28.582ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
14442024-07-25T05:45:28.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
14452024-07-25T05:45:28.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
14462024-07-25T05:45:28.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
14472024-07-25T05:45:28.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
14482024-07-25T05:45:28.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
14492024-07-25T05:45:28.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
14502024-07-25T05:45:28.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
14512024-07-25T05:45:28.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
14522024-07-25T05:45:28.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
14532024-07-25T05:45:28.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
14542024-07-25T05:45:28.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
14552024-07-25T05:45:28.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
14562024-07-25T05:45:28.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
14572024-07-25T05:45:28.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
14582024-07-25T05:45:28.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
14592024-07-25T05:45:28.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
14602024-07-25T05:45:28.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
14612024-07-25T05:45:28.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
14622024-07-25T05:45:28.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
14632024-07-25T05:45:28.678ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3", TimeNotSynchronized), ("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c", TimeNotSynchronized), ("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d", TimeNotSynchronized), ("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c", TimeNotSynchronized), ("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862", TimeNotSynchronized), ("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa", TimeNotSynchronized), ("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046", TimeNotSynchronized), ("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7", TimeNotSynchronized), ("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480", TimeNotSynchronized), ("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876", TimeNotSynchronized), ("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465", TimeNotSynchronized), ("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b", TimeNotSynchronized), ("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095", TimeNotSynchronized), ("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a", TimeNotSynchronized), ("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e", TimeNotSynchronized), ("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2", TimeNotSynchronized), ("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc", TimeNotSynchronized), ("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9", TimeNotSynchronized), ("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 504354 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:47495 req_id = 3bde4b7e-0be2-4386-a176-a3150fa8f837 response_code = 500 uri = /omicron-zones
14642024-07-25T05:45:28.678ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "3bde4b7e-0be2-4386-a176-a3150fa8f837", "content-length": "124", "date": "Thu, 25 Jul 2024 05:45:27 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "3bde4b7e-0be2-4386-a176-a3150fa8f837" } file = sled-agent/src/rack_setup/service.rs:472 retry_after = 751.935821ms sled_address = [fd00:1122:3344:101::1]:12345
14652024-07-25T05:45:29.433ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6389ba5c-9d50-4fe4-b1a2-8be3c6861cb0, kind: Service { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c }, name: Name("ntp-3f311cd5-ee38-4707-ab8a-9331491f8e7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 235])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 4f6a29ee-ca23-4fa8-be69-497060121028, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: b04db43b-dc24-43ce-9d97-32193f88e046, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 2c3f8453-233f-428f-93cc-0bfefb84929d, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: e4efbc31-e513-4290-82f5-c51d8afea0c7, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 221c5e7c-0832-4890-8b0a-f9ba08725876, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4b4998f4-10de-49f2-9326-b482137c5fd3, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: a635cc62-7516-4fd9-b490-62d6de386095, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ac9fa851-e7e1-4295-ac79-d6076526f465, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 1cb96976-2047-44b8-a79c-9fcd74d17bfa, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: a0b9fba0-c330-402f-9bc0-a7510786c98a, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d94d9602-d610-44ce-b9fc-06ded41a1480, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb1f626-0f55-4385-87de-de59052d98d2, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External } } } }, OmicronZoneConfig { id: adff3617-5320-4e42-8b10-1430a1eb0afc, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:434 sled_address = [fd00:1122:3344:101::1]:12345
14662024-07-25T05:45:29.434ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
14672024-07-25T05:45:29.434ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }
14682024-07-25T05:45:29.458ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
14692024-07-25T05:45:29.458ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }
14702024-07-25T05:45:29.478ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
14712024-07-25T05:45:29.478ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14722024-07-25T05:45:29.497ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
14732024-07-25T05:45:29.497ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14742024-07-25T05:45:29.516ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
14752024-07-25T05:45:29.516ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14762024-07-25T05:45:29.538ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
14772024-07-25T05:45:29.538ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14782024-07-25T05:45:29.558ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
14792024-07-25T05:45:29.558ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14802024-07-25T05:45:29.576ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
14812024-07-25T05:45:29.576ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14822024-07-25T05:45:29.596ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
14832024-07-25T05:45:29.596ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }
14842024-07-25T05:45:29.616ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
14852024-07-25T05:45:29.616ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }
14862024-07-25T05:45:29.637ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
14872024-07-25T05:45:29.637ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }
14882024-07-25T05:45:29.657ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
14892024-07-25T05:45:29.657ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }
14902024-07-25T05:45:29.676ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
14912024-07-25T05:45:29.676ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }
14922024-07-25T05:45:29.694ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
14932024-07-25T05:45:29.694ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }
14942024-07-25T05:45:29.715ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
14952024-07-25T05:45:29.715ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }
14962024-07-25T05:45:29.734ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
14972024-07-25T05:45:29.734ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }
14982024-07-25T05:45:29.754ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
14992024-07-25T05:45:29.754ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }
15002024-07-25T05:45:29.773ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
15012024-07-25T05:45:29.774ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }
15022024-07-25T05:45:29.794ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
15032024-07-25T05:45:29.794ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }
15042024-07-25T05:45:29.813ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
15052024-07-25T05:45:29.813ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }
15062024-07-25T05:45:29.834ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
15072024-07-25T05:45:29.834ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:177
15082024-07-25T05:45:29.834ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:177
15092024-07-25T05:45:29.834ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
15102024-07-25T05:45:29.931ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
15112024-07-25T05:45:29.931ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
15122024-07-25T05:45:29.931ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
15132024-07-25T05:45:29.931ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
15142024-07-25T05:45:29.931ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
15152024-07-25T05:45:29.931ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
15162024-07-25T05:45:29.931ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
15172024-07-25T05:45:29.931ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
15182024-07-25T05:45:29.932ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
15192024-07-25T05:45:29.932ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
15202024-07-25T05:45:29.932ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
15212024-07-25T05:45:29.932ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
15222024-07-25T05:45:29.932ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
15232024-07-25T05:45:29.932ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
15242024-07-25T05:45:29.932ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
15252024-07-25T05:45:29.932ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
15262024-07-25T05:45:29.932ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
15272024-07-25T05:45:29.932ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
15282024-07-25T05:45:29.932ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
15292024-07-25T05:45:29.932ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7", TimeNotSynchronized), ("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876", TimeNotSynchronized), ("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046", TimeNotSynchronized), ("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862", TimeNotSynchronized), ("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480", TimeNotSynchronized), ("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a", TimeNotSynchronized), ("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c", TimeNotSynchronized), ("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9", TimeNotSynchronized), ("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b", TimeNotSynchronized), ("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095", TimeNotSynchronized), ("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79", TimeNotSynchronized), ("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e", TimeNotSynchronized), ("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d", TimeNotSynchronized), ("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc", TimeNotSynchronized), ("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c", TimeNotSynchronized), ("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa", TimeNotSynchronized), ("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2", TimeNotSynchronized), ("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3", TimeNotSynchronized), ("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 496875 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:47495 req_id = 8ca8134e-3eed-487a-93eb-ed5cb8048079 response_code = 500 uri = /omicron-zones
15302024-07-25T05:45:29.932ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "8ca8134e-3eed-487a-93eb-ed5cb8048079", "content-length": "124", "date": "Thu, 25 Jul 2024 05:45:29 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "8ca8134e-3eed-487a-93eb-ed5cb8048079" } file = sled-agent/src/rack_setup/service.rs:472 retry_after = 449.825485ms sled_address = [fd00:1122:3344:101::1]:12345
15312024-07-25T05:45:30.384ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6389ba5c-9d50-4fe4-b1a2-8be3c6861cb0, kind: Service { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c }, name: Name("ntp-3f311cd5-ee38-4707-ab8a-9331491f8e7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 235])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 4f6a29ee-ca23-4fa8-be69-497060121028, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: b04db43b-dc24-43ce-9d97-32193f88e046, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 2c3f8453-233f-428f-93cc-0bfefb84929d, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: e4efbc31-e513-4290-82f5-c51d8afea0c7, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 221c5e7c-0832-4890-8b0a-f9ba08725876, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4b4998f4-10de-49f2-9326-b482137c5fd3, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: a635cc62-7516-4fd9-b490-62d6de386095, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ac9fa851-e7e1-4295-ac79-d6076526f465, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 1cb96976-2047-44b8-a79c-9fcd74d17bfa, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: a0b9fba0-c330-402f-9bc0-a7510786c98a, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d94d9602-d610-44ce-b9fc-06ded41a1480, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb1f626-0f55-4385-87de-de59052d98d2, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External } } } }, OmicronZoneConfig { id: adff3617-5320-4e42-8b10-1430a1eb0afc, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:434 sled_address = [fd00:1122:3344:101::1]:12345
15322024-07-25T05:45:30.384ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
15332024-07-25T05:45:30.384ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }
15342024-07-25T05:45:30.408ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
15352024-07-25T05:45:30.408ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }
15362024-07-25T05:45:30.430ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
15372024-07-25T05:45:30.430ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15382024-07-25T05:45:30.451ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
15392024-07-25T05:45:30.451ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15402024-07-25T05:45:30.471ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
15412024-07-25T05:45:30.471ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15422024-07-25T05:45:30.491ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
15432024-07-25T05:45:30.491ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15442024-07-25T05:45:30.510ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
15452024-07-25T05:45:30.511ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15462024-07-25T05:45:30.531ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
15472024-07-25T05:45:30.531ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15482024-07-25T05:45:30.550ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
15492024-07-25T05:45:30.550ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }
15502024-07-25T05:45:30.570ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
15512024-07-25T05:45:30.570ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }
15522024-07-25T05:45:30.591ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
15532024-07-25T05:45:30.591ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }
15542024-07-25T05:45:30.610ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
15552024-07-25T05:45:30.610ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }
15562024-07-25T05:45:30.630ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
15572024-07-25T05:45:30.630ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }
15582024-07-25T05:45:30.650ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
15592024-07-25T05:45:30.650ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }
15602024-07-25T05:45:30.670ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
15612024-07-25T05:45:30.670ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }
15622024-07-25T05:45:30.692ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
15632024-07-25T05:45:30.692ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }
15642024-07-25T05:45:30.712ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
15652024-07-25T05:45:30.712ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }
15662024-07-25T05:45:30.732ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
15672024-07-25T05:45:30.732ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }
15682024-07-25T05:45:30.752ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
15692024-07-25T05:45:30.752ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }
15702024-07-25T05:45:30.771ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
15712024-07-25T05:45:30.771ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }
15722024-07-25T05:45:30.791ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
15732024-07-25T05:45:30.791ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:177
15742024-07-25T05:45:30.792ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:177
15752024-07-25T05:45:30.792ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
15762024-07-25T05:45:30.891ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
15772024-07-25T05:45:30.891ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
15782024-07-25T05:45:30.891ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
15792024-07-25T05:45:30.891ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
15802024-07-25T05:45:30.891ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
15812024-07-25T05:45:30.891ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
15822024-07-25T05:45:30.891ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
15832024-07-25T05:45:30.891ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
15842024-07-25T05:45:30.891ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
15852024-07-25T05:45:30.891ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
15862024-07-25T05:45:30.891ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
15872024-07-25T05:45:30.891ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
15882024-07-25T05:45:30.891ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
15892024-07-25T05:45:30.891ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
15902024-07-25T05:45:30.891ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
15912024-07-25T05:45:30.891ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
15922024-07-25T05:45:30.891ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
15932024-07-25T05:45:30.891ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
15942024-07-25T05:45:30.891ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
15952024-07-25T05:45:30.891ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79", TimeNotSynchronized), ("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9", TimeNotSynchronized), ("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046", TimeNotSynchronized), ("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a", TimeNotSynchronized), ("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480", TimeNotSynchronized), ("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876", TimeNotSynchronized), ("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2", TimeNotSynchronized), ("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa", TimeNotSynchronized), ("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc", TimeNotSynchronized), ("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465", TimeNotSynchronized), ("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c", TimeNotSynchronized), ("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7", TimeNotSynchronized), ("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862", TimeNotSynchronized), ("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c", TimeNotSynchronized), ("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b", TimeNotSynchronized), ("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3", TimeNotSynchronized), ("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095", TimeNotSynchronized), ("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e", TimeNotSynchronized), ("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 506272 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:47495 req_id = aa716d32-6c05-4329-9828-4353f98bf632 response_code = 500 uri = /omicron-zones
15962024-07-25T05:45:30.892ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "aa716d32-6c05-4329-9828-4353f98bf632", "content-length": "124", "date": "Thu, 25 Jul 2024 05:45:30 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "aa716d32-6c05-4329-9828-4353f98bf632" } file = sled-agent/src/rack_setup/service.rs:472 retry_after = 815.289752ms sled_address = [fd00:1122:3344:101::1]:12345
15972024-07-25T05:45:31.710ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6389ba5c-9d50-4fe4-b1a2-8be3c6861cb0, kind: Service { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c }, name: Name("ntp-3f311cd5-ee38-4707-ab8a-9331491f8e7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 235])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 4f6a29ee-ca23-4fa8-be69-497060121028, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: b04db43b-dc24-43ce-9d97-32193f88e046, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 2c3f8453-233f-428f-93cc-0bfefb84929d, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: e4efbc31-e513-4290-82f5-c51d8afea0c7, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 221c5e7c-0832-4890-8b0a-f9ba08725876, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4b4998f4-10de-49f2-9326-b482137c5fd3, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: a635cc62-7516-4fd9-b490-62d6de386095, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ac9fa851-e7e1-4295-ac79-d6076526f465, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 1cb96976-2047-44b8-a79c-9fcd74d17bfa, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: a0b9fba0-c330-402f-9bc0-a7510786c98a, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d94d9602-d610-44ce-b9fc-06ded41a1480, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb1f626-0f55-4385-87de-de59052d98d2, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External } } } }, OmicronZoneConfig { id: adff3617-5320-4e42-8b10-1430a1eb0afc, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:434 sled_address = [fd00:1122:3344:101::1]:12345
15982024-07-25T05:45:31.711ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
15992024-07-25T05:45:31.711ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }
16002024-07-25T05:45:31.736ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
16012024-07-25T05:45:31.736ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }
16022024-07-25T05:45:31.758ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
16032024-07-25T05:45:31.758ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }
16042024-07-25T05:45:31.778ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
16052024-07-25T05:45:31.778ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16062024-07-25T05:45:31.796ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
16072024-07-25T05:45:31.796ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16082024-07-25T05:45:31.816ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
16092024-07-25T05:45:31.817ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16102024-07-25T05:45:31.835ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
16112024-07-25T05:45:31.835ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16122024-07-25T05:45:31.854ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
16132024-07-25T05:45:31.854ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16142024-07-25T05:45:31.873ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
16152024-07-25T05:45:31.873ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }
16162024-07-25T05:45:31.892ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
16172024-07-25T05:45:31.892ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }
16182024-07-25T05:45:31.910ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
16192024-07-25T05:45:31.910ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }
16202024-07-25T05:45:31.932ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
16212024-07-25T05:45:31.932ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }
16222024-07-25T05:45:31.953ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
16232024-07-25T05:45:31.953ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }
16242024-07-25T05:45:31.972ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
16252024-07-25T05:45:31.972ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }
16262024-07-25T05:45:31.992ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
16272024-07-25T05:45:31.992ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }
16282024-07-25T05:45:32.011ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
16292024-07-25T05:45:32.011ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }
16302024-07-25T05:45:32.033ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
16312024-07-25T05:45:32.033ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }
16322024-07-25T05:45:32.054ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
16332024-07-25T05:45:32.054ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }
16342024-07-25T05:45:32.072ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
16352024-07-25T05:45:32.072ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }
16362024-07-25T05:45:32.092ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
16372024-07-25T05:45:32.092ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }
16382024-07-25T05:45:32.114ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
16392024-07-25T05:45:32.114ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:177
16402024-07-25T05:45:32.114ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:177
16412024-07-25T05:45:32.114ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
16422024-07-25T05:45:32.218ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
16432024-07-25T05:45:32.218ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
16442024-07-25T05:45:32.218ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
16452024-07-25T05:45:32.218ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
16462024-07-25T05:45:32.218ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
16472024-07-25T05:45:32.218ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
16482024-07-25T05:45:32.218ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
16492024-07-25T05:45:32.218ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
16502024-07-25T05:45:32.218ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
16512024-07-25T05:45:32.218ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
16522024-07-25T05:45:32.218ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
16532024-07-25T05:45:32.218ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
16542024-07-25T05:45:32.218ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
16552024-07-25T05:45:32.218ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
16562024-07-25T05:45:32.218ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
16572024-07-25T05:45:32.218ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
16582024-07-25T05:45:32.218ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
16592024-07-25T05:45:32.218ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
16602024-07-25T05:45:32.218ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
16612024-07-25T05:45:32.218ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876", TimeNotSynchronized), ("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c", TimeNotSynchronized), ("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b", TimeNotSynchronized), ("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3", TimeNotSynchronized), ("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c", TimeNotSynchronized), ("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa", TimeNotSynchronized), ("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480", TimeNotSynchronized), ("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e", TimeNotSynchronized), ("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862", TimeNotSynchronized), ("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095", TimeNotSynchronized), ("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465", TimeNotSynchronized), ("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046", TimeNotSynchronized), ("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d", TimeNotSynchronized), ("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7", TimeNotSynchronized), ("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a", TimeNotSynchronized), ("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc", TimeNotSynchronized), ("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9", TimeNotSynchronized), ("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79", TimeNotSynchronized), ("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 506424 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:47495 req_id = 30e8eb38-8981-4f11-a681-134c44b08a3e response_code = 500 uri = /omicron-zones
16622024-07-25T05:45:32.219ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "30e8eb38-8981-4f11-a681-134c44b08a3e", "content-length": "124", "date": "Thu, 25 Jul 2024 05:45:32 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "30e8eb38-8981-4f11-a681-134c44b08a3e" } file = sled-agent/src/rack_setup/service.rs:472 retry_after = 1.224533693s sled_address = [fd00:1122:3344:101::1]:12345
16632024-07-25T05:45:33.448ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6389ba5c-9d50-4fe4-b1a2-8be3c6861cb0, kind: Service { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c }, name: Name("ntp-3f311cd5-ee38-4707-ab8a-9331491f8e7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 235])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 4f6a29ee-ca23-4fa8-be69-497060121028, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: b04db43b-dc24-43ce-9d97-32193f88e046, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 2c3f8453-233f-428f-93cc-0bfefb84929d, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: e4efbc31-e513-4290-82f5-c51d8afea0c7, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 221c5e7c-0832-4890-8b0a-f9ba08725876, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4b4998f4-10de-49f2-9326-b482137c5fd3, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: a635cc62-7516-4fd9-b490-62d6de386095, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ac9fa851-e7e1-4295-ac79-d6076526f465, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 1cb96976-2047-44b8-a79c-9fcd74d17bfa, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: a0b9fba0-c330-402f-9bc0-a7510786c98a, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d94d9602-d610-44ce-b9fc-06ded41a1480, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb1f626-0f55-4385-87de-de59052d98d2, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External } } } }, OmicronZoneConfig { id: adff3617-5320-4e42-8b10-1430a1eb0afc, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:434 sled_address = [fd00:1122:3344:101::1]:12345
16642024-07-25T05:45:33.448ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
16652024-07-25T05:45:33.448ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }
16662024-07-25T05:45:33.471ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
16672024-07-25T05:45:33.471ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }
16682024-07-25T05:45:33.493ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
16692024-07-25T05:45:33.493ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }
16702024-07-25T05:45:33.517ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
16712024-07-25T05:45:33.517ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16722024-07-25T05:45:33.538ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
16732024-07-25T05:45:33.538ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16742024-07-25T05:45:33.559ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
16752024-07-25T05:45:33.559ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16762024-07-25T05:45:33.579ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
16772024-07-25T05:45:33.579ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16782024-07-25T05:45:33.600ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
16792024-07-25T05:45:33.600ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16802024-07-25T05:45:33.621ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
16812024-07-25T05:45:33.621ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }
16822024-07-25T05:45:33.639ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
16832024-07-25T05:45:33.639ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }
16842024-07-25T05:45:33.660ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
16852024-07-25T05:45:33.660ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }
16862024-07-25T05:45:33.680ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
16872024-07-25T05:45:33.680ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }
16882024-07-25T05:45:33.701ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
16892024-07-25T05:45:33.701ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }
16902024-07-25T05:45:33.722ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
16912024-07-25T05:45:33.722ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }
16922024-07-25T05:45:33.743ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
16932024-07-25T05:45:33.743ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }
16942024-07-25T05:45:33.762ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
16952024-07-25T05:45:33.762ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }
16962024-07-25T05:45:33.784ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
16972024-07-25T05:45:33.784ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }
16982024-07-25T05:45:33.803ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
16992024-07-25T05:45:33.803ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }
17002024-07-25T05:45:33.824ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
17012024-07-25T05:45:33.824ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }
17022024-07-25T05:45:33.845ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
17032024-07-25T05:45:33.845ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }
17042024-07-25T05:45:33.865ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
17052024-07-25T05:45:33.865ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:177
17062024-07-25T05:45:33.866ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:177
17072024-07-25T05:45:33.866ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
17082024-07-25T05:45:33.963ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
17092024-07-25T05:45:33.963ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
17102024-07-25T05:45:33.963ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
17112024-07-25T05:45:33.963ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
17122024-07-25T05:45:33.963ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
17132024-07-25T05:45:33.963ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
17142024-07-25T05:45:33.963ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
17152024-07-25T05:45:33.963ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
17162024-07-25T05:45:33.963ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
17172024-07-25T05:45:33.963ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
17182024-07-25T05:45:33.963ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
17192024-07-25T05:45:33.963ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
17202024-07-25T05:45:33.963ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
17212024-07-25T05:45:33.963ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
17222024-07-25T05:45:33.963ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
17232024-07-25T05:45:33.963ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
17242024-07-25T05:45:33.963ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
17252024-07-25T05:45:33.963ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
17262024-07-25T05:45:33.963ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
17272024-07-25T05:45:33.963ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3", TimeNotSynchronized), ("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc", TimeNotSynchronized), ("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095", TimeNotSynchronized), ("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876", TimeNotSynchronized), ("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862", TimeNotSynchronized), ("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046", TimeNotSynchronized), ("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465", TimeNotSynchronized), ("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa", TimeNotSynchronized), ("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e", TimeNotSynchronized), ("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7", TimeNotSynchronized), ("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2", TimeNotSynchronized), ("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c", TimeNotSynchronized), ("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480", TimeNotSynchronized), ("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79", TimeNotSynchronized), ("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b", TimeNotSynchronized), ("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a", TimeNotSynchronized), ("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d", TimeNotSynchronized), ("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c", TimeNotSynchronized), ("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 514091 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:47495 req_id = cb26b278-0bca-494c-8b86-9c68cc618ee9 response_code = 500 uri = /omicron-zones
17282024-07-25T05:45:33.963ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "cb26b278-0bca-494c-8b86-9c68cc618ee9", "content-length": "124", "date": "Thu, 25 Jul 2024 05:45:33 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "cb26b278-0bca-494c-8b86-9c68cc618ee9" } file = sled-agent/src/rack_setup/service.rs:472 retry_after = 771.103491ms sled_address = [fd00:1122:3344:101::1]:12345
17292024-07-25T05:45:34.738ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6389ba5c-9d50-4fe4-b1a2-8be3c6861cb0, kind: Service { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c }, name: Name("ntp-3f311cd5-ee38-4707-ab8a-9331491f8e7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 235])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 4f6a29ee-ca23-4fa8-be69-497060121028, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: b04db43b-dc24-43ce-9d97-32193f88e046, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 2c3f8453-233f-428f-93cc-0bfefb84929d, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: e4efbc31-e513-4290-82f5-c51d8afea0c7, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 221c5e7c-0832-4890-8b0a-f9ba08725876, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4b4998f4-10de-49f2-9326-b482137c5fd3, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: a635cc62-7516-4fd9-b490-62d6de386095, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ac9fa851-e7e1-4295-ac79-d6076526f465, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 1cb96976-2047-44b8-a79c-9fcd74d17bfa, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: a0b9fba0-c330-402f-9bc0-a7510786c98a, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d94d9602-d610-44ce-b9fc-06ded41a1480, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb1f626-0f55-4385-87de-de59052d98d2, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External } } } }, OmicronZoneConfig { id: adff3617-5320-4e42-8b10-1430a1eb0afc, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:434 sled_address = [fd00:1122:3344:101::1]:12345
17302024-07-25T05:45:34.738ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
17312024-07-25T05:45:34.738ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }
17322024-07-25T05:45:34.762ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
17332024-07-25T05:45:34.762ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }
17342024-07-25T05:45:34.782ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
17352024-07-25T05:45:34.782ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }
17362024-07-25T05:45:34.802ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
17372024-07-25T05:45:34.802ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17382024-07-25T05:45:34.822ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
17392024-07-25T05:45:34.822ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17402024-07-25T05:45:34.843ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
17412024-07-25T05:45:34.843ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17422024-07-25T05:45:34.863ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
17432024-07-25T05:45:34.863ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17442024-07-25T05:45:34.881ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
17452024-07-25T05:45:34.881ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17462024-07-25T05:45:34.901ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
17472024-07-25T05:45:34.901ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }
17482024-07-25T05:45:34.921ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
17492024-07-25T05:45:34.921ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }
17502024-07-25T05:45:34.941ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
17512024-07-25T05:45:34.941ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }
17522024-07-25T05:45:34.963ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
17532024-07-25T05:45:34.963ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }
17542024-07-25T05:45:34.982ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
17552024-07-25T05:45:34.982ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }
17562024-07-25T05:45:35.002ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
17572024-07-25T05:45:35.002ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }
17582024-07-25T05:45:35.024ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
17592024-07-25T05:45:35.024ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }
17602024-07-25T05:45:35.052ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
17612024-07-25T05:45:35.052ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }
17622024-07-25T05:45:35.072ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
17632024-07-25T05:45:35.072ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }
17642024-07-25T05:45:35.091ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
17652024-07-25T05:45:35.091ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }
17662024-07-25T05:45:35.110ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
17672024-07-25T05:45:35.110ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }
17682024-07-25T05:45:35.129ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
17692024-07-25T05:45:35.129ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }
17702024-07-25T05:45:35.148ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
17712024-07-25T05:45:35.148ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:177
17722024-07-25T05:45:35.148ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:177
17732024-07-25T05:45:35.148ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
17742024-07-25T05:45:35.248ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
17752024-07-25T05:45:35.248ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
17762024-07-25T05:45:35.248ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
17772024-07-25T05:45:35.248ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
17782024-07-25T05:45:35.248ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
17792024-07-25T05:45:35.248ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
17802024-07-25T05:45:35.248ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
17812024-07-25T05:45:35.248ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
17822024-07-25T05:45:35.248ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
17832024-07-25T05:45:35.248ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
17842024-07-25T05:45:35.248ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
17852024-07-25T05:45:35.248ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
17862024-07-25T05:45:35.248ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
17872024-07-25T05:45:35.248ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
17882024-07-25T05:45:35.248ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
17892024-07-25T05:45:35.248ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
17902024-07-25T05:45:35.248ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
17912024-07-25T05:45:35.248ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
17922024-07-25T05:45:35.248ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
17932024-07-25T05:45:35.248ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa", TimeNotSynchronized), ("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480", TimeNotSynchronized), ("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465", TimeNotSynchronized), ("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e", TimeNotSynchronized), ("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9", TimeNotSynchronized), ("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862", TimeNotSynchronized), ("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d", TimeNotSynchronized), ("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046", TimeNotSynchronized), ("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3", TimeNotSynchronized), ("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c", TimeNotSynchronized), ("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a", TimeNotSynchronized), ("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7", TimeNotSynchronized), ("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc", TimeNotSynchronized), ("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095", TimeNotSynchronized), ("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b", TimeNotSynchronized), ("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876", TimeNotSynchronized), ("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c", TimeNotSynchronized), ("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79", TimeNotSynchronized), ("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 508799 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:47495 req_id = f2a59ba6-96e6-4452-bf86-c93c2c12425d response_code = 500 uri = /omicron-zones
17942024-07-25T05:45:35.249ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "f2a59ba6-96e6-4452-bf86-c93c2c12425d", "content-length": "124", "date": "Thu, 25 Jul 2024 05:45:34 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "f2a59ba6-96e6-4452-bf86-c93c2c12425d" } file = sled-agent/src/rack_setup/service.rs:472 retry_after = 1.308676473s sled_address = [fd00:1122:3344:101::1]:12345
17952024-07-25T05:45:36.562ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6389ba5c-9d50-4fe4-b1a2-8be3c6861cb0, kind: Service { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c }, name: Name("ntp-3f311cd5-ee38-4707-ab8a-9331491f8e7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 235])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 4f6a29ee-ca23-4fa8-be69-497060121028, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: b04db43b-dc24-43ce-9d97-32193f88e046, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 2c3f8453-233f-428f-93cc-0bfefb84929d, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: e4efbc31-e513-4290-82f5-c51d8afea0c7, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 221c5e7c-0832-4890-8b0a-f9ba08725876, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4b4998f4-10de-49f2-9326-b482137c5fd3, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: a635cc62-7516-4fd9-b490-62d6de386095, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ac9fa851-e7e1-4295-ac79-d6076526f465, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 1cb96976-2047-44b8-a79c-9fcd74d17bfa, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: a0b9fba0-c330-402f-9bc0-a7510786c98a, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d94d9602-d610-44ce-b9fc-06ded41a1480, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb1f626-0f55-4385-87de-de59052d98d2, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External } } } }, OmicronZoneConfig { id: adff3617-5320-4e42-8b10-1430a1eb0afc, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:434 sled_address = [fd00:1122:3344:101::1]:12345
17962024-07-25T05:45:36.563ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
17972024-07-25T05:45:36.563ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }
17982024-07-25T05:45:36.586ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
17992024-07-25T05:45:36.586ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }
18002024-07-25T05:45:36.607ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
18012024-07-25T05:45:36.608ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }
18022024-07-25T05:45:36.628ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
18032024-07-25T05:45:36.628ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18042024-07-25T05:45:36.648ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
18052024-07-25T05:45:36.648ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18062024-07-25T05:45:36.668ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
18072024-07-25T05:45:36.668ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18082024-07-25T05:45:36.687ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
18092024-07-25T05:45:36.687ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18102024-07-25T05:45:36.707ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
18112024-07-25T05:45:36.707ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18122024-07-25T05:45:36.728ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
18132024-07-25T05:45:36.728ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }
18142024-07-25T05:45:36.749ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
18152024-07-25T05:45:36.749ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }
18162024-07-25T05:45:36.770ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
18172024-07-25T05:45:36.770ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }
18182024-07-25T05:45:36.791ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
18192024-07-25T05:45:36.791ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }
18202024-07-25T05:45:36.813ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
18212024-07-25T05:45:36.813ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }
18222024-07-25T05:45:36.833ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
18232024-07-25T05:45:36.833ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }
18242024-07-25T05:45:36.850ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 1.946912249s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
18252024-07-25T05:45:36.855ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
18262024-07-25T05:45:36.855ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }
18272024-07-25T05:45:36.877ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
18282024-07-25T05:45:36.877ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }
18292024-07-25T05:45:36.899ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
18302024-07-25T05:45:36.899ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }
18312024-07-25T05:45:36.920ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
18322024-07-25T05:45:36.920ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }
18332024-07-25T05:45:36.940ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
18342024-07-25T05:45:36.940ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }
18352024-07-25T05:45:36.962ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
18362024-07-25T05:45:36.962ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }
18372024-07-25T05:45:36.983ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
18382024-07-25T05:45:36.983ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:177
18392024-07-25T05:45:36.983ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:177
18402024-07-25T05:45:36.983ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
18412024-07-25T05:45:37.085ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
18422024-07-25T05:45:37.085ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
18432024-07-25T05:45:37.085ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
18442024-07-25T05:45:37.085ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
18452024-07-25T05:45:37.085ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
18462024-07-25T05:45:37.085ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
18472024-07-25T05:45:37.085ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
18482024-07-25T05:45:37.085ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
18492024-07-25T05:45:37.085ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
18502024-07-25T05:45:37.085ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
18512024-07-25T05:45:37.085ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
18522024-07-25T05:45:37.085ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
18532024-07-25T05:45:37.085ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
18542024-07-25T05:45:37.085ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
18552024-07-25T05:45:37.085ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
18562024-07-25T05:45:37.085ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
18572024-07-25T05:45:37.085ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
18582024-07-25T05:45:37.085ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
18592024-07-25T05:45:37.085ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
18602024-07-25T05:45:37.085ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7", TimeNotSynchronized), ("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b", TimeNotSynchronized), ("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095", TimeNotSynchronized), ("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465", TimeNotSynchronized), ("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d", TimeNotSynchronized), ("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a", TimeNotSynchronized), ("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c", TimeNotSynchronized), ("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa", TimeNotSynchronized), ("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480", TimeNotSynchronized), ("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79", TimeNotSynchronized), ("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046", TimeNotSynchronized), ("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e", TimeNotSynchronized), ("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c", TimeNotSynchronized), ("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2", TimeNotSynchronized), ("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc", TimeNotSynchronized), ("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9", TimeNotSynchronized), ("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3", TimeNotSynchronized), ("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876", TimeNotSynchronized), ("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 521566 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:47495 req_id = aa88a932-5faf-4460-b201-1e736dd2fd18 response_code = 500 uri = /omicron-zones
18612024-07-25T05:45:37.085ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "aa88a932-5faf-4460-b201-1e736dd2fd18", "content-length": "124", "date": "Thu, 25 Jul 2024 05:45:36 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "aa88a932-5faf-4460-b201-1e736dd2fd18" } file = sled-agent/src/rack_setup/service.rs:472 retry_after = 1.325593067s sled_address = [fd00:1122:3344:101::1]:12345
18622024-07-25T05:45:38.416ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6389ba5c-9d50-4fe4-b1a2-8be3c6861cb0, kind: Service { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c }, name: Name("ntp-3f311cd5-ee38-4707-ab8a-9331491f8e7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 235])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 4f6a29ee-ca23-4fa8-be69-497060121028, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: b04db43b-dc24-43ce-9d97-32193f88e046, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 2c3f8453-233f-428f-93cc-0bfefb84929d, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: e4efbc31-e513-4290-82f5-c51d8afea0c7, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 221c5e7c-0832-4890-8b0a-f9ba08725876, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4b4998f4-10de-49f2-9326-b482137c5fd3, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: a635cc62-7516-4fd9-b490-62d6de386095, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ac9fa851-e7e1-4295-ac79-d6076526f465, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 1cb96976-2047-44b8-a79c-9fcd74d17bfa, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: a0b9fba0-c330-402f-9bc0-a7510786c98a, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d94d9602-d610-44ce-b9fc-06ded41a1480, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb1f626-0f55-4385-87de-de59052d98d2, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External } } } }, OmicronZoneConfig { id: adff3617-5320-4e42-8b10-1430a1eb0afc, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:434 sled_address = [fd00:1122:3344:101::1]:12345
18632024-07-25T05:45:38.417ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
18642024-07-25T05:45:38.417ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }
18652024-07-25T05:45:38.439ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
18662024-07-25T05:45:38.439ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }
18672024-07-25T05:45:38.460ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
18682024-07-25T05:45:38.460ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }
18692024-07-25T05:45:38.479ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
18702024-07-25T05:45:38.479ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18712024-07-25T05:45:38.501ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
18722024-07-25T05:45:38.501ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18732024-07-25T05:45:38.523ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
18742024-07-25T05:45:38.523ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18752024-07-25T05:45:38.545ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
18762024-07-25T05:45:38.545ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18772024-07-25T05:45:38.567ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
18782024-07-25T05:45:38.567ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18792024-07-25T05:45:38.590ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
18802024-07-25T05:45:38.590ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }
18812024-07-25T05:45:38.611ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
18822024-07-25T05:45:38.611ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }
18832024-07-25T05:45:38.632ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
18842024-07-25T05:45:38.632ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }
18852024-07-25T05:45:38.653ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
18862024-07-25T05:45:38.653ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }
18872024-07-25T05:45:38.676ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
18882024-07-25T05:45:38.676ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }
18892024-07-25T05:45:38.699ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
18902024-07-25T05:45:38.699ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }
18912024-07-25T05:45:38.721ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
18922024-07-25T05:45:38.721ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }
18932024-07-25T05:45:38.743ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
18942024-07-25T05:45:38.743ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }
18952024-07-25T05:45:38.764ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
18962024-07-25T05:45:38.764ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }
18972024-07-25T05:45:38.785ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
18982024-07-25T05:45:38.785ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }
18992024-07-25T05:45:38.806ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
19002024-07-25T05:45:38.806ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }
19012024-07-25T05:45:38.828ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
19022024-07-25T05:45:38.828ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }
19032024-07-25T05:45:38.850ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
19042024-07-25T05:45:38.850ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:177
19052024-07-25T05:45:38.851ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:177
19062024-07-25T05:45:38.851ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
19072024-07-25T05:45:38.952ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
19082024-07-25T05:45:38.952ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
19092024-07-25T05:45:38.952ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
19102024-07-25T05:45:38.952ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
19112024-07-25T05:45:38.952ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
19122024-07-25T05:45:38.952ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
19132024-07-25T05:45:38.952ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
19142024-07-25T05:45:38.952ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
19152024-07-25T05:45:38.952ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
19162024-07-25T05:45:38.952ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
19172024-07-25T05:45:38.952ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
19182024-07-25T05:45:38.952ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
19192024-07-25T05:45:38.952ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
19202024-07-25T05:45:38.952ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
19212024-07-25T05:45:38.952ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
19222024-07-25T05:45:38.952ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
19232024-07-25T05:45:38.952ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
19242024-07-25T05:45:38.952ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
19252024-07-25T05:45:38.952ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
19262024-07-25T05:45:38.952ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3", TimeNotSynchronized), ("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465", TimeNotSynchronized), ("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa", TimeNotSynchronized), ("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc", TimeNotSynchronized), ("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9", TimeNotSynchronized), ("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480", TimeNotSynchronized), ("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79", TimeNotSynchronized), ("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046", TimeNotSynchronized), ("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b", TimeNotSynchronized), ("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a", TimeNotSynchronized), ("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c", TimeNotSynchronized), ("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876", TimeNotSynchronized), ("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c", TimeNotSynchronized), ("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d", TimeNotSynchronized), ("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e", TimeNotSynchronized), ("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2", TimeNotSynchronized), ("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7", TimeNotSynchronized), ("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862", TimeNotSynchronized), ("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 534864 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:47495 req_id = 8eb183c4-cd58-4a8a-81d4-67679171915a response_code = 500 uri = /omicron-zones
19272024-07-25T05:45:38.953ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "8eb183c4-cd58-4a8a-81d4-67679171915a", "content-length": "124", "date": "Thu, 25 Jul 2024 05:45:38 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "8eb183c4-cd58-4a8a-81d4-67679171915a" } file = sled-agent/src/rack_setup/service.rs:472 retry_after = 2.307591668s sled_address = [fd00:1122:3344:101::1]:12345
19282024-07-25T05:45:41.268ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6389ba5c-9d50-4fe4-b1a2-8be3c6861cb0, kind: Service { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c }, name: Name("ntp-3f311cd5-ee38-4707-ab8a-9331491f8e7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 235])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 4f6a29ee-ca23-4fa8-be69-497060121028, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: b04db43b-dc24-43ce-9d97-32193f88e046, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 2c3f8453-233f-428f-93cc-0bfefb84929d, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: e4efbc31-e513-4290-82f5-c51d8afea0c7, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 221c5e7c-0832-4890-8b0a-f9ba08725876, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4b4998f4-10de-49f2-9326-b482137c5fd3, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: a635cc62-7516-4fd9-b490-62d6de386095, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ac9fa851-e7e1-4295-ac79-d6076526f465, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 1cb96976-2047-44b8-a79c-9fcd74d17bfa, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: a0b9fba0-c330-402f-9bc0-a7510786c98a, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d94d9602-d610-44ce-b9fc-06ded41a1480, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb1f626-0f55-4385-87de-de59052d98d2, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External } } } }, OmicronZoneConfig { id: adff3617-5320-4e42-8b10-1430a1eb0afc, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:434 sled_address = [fd00:1122:3344:101::1]:12345
19292024-07-25T05:45:41.268ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
19302024-07-25T05:45:41.269ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }
19312024-07-25T05:45:41.291ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
19322024-07-25T05:45:41.292ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }
19332024-07-25T05:45:41.312ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
19342024-07-25T05:45:41.312ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }
19352024-07-25T05:45:41.333ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
19362024-07-25T05:45:41.333ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19372024-07-25T05:45:41.351ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
19382024-07-25T05:45:41.352ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19392024-07-25T05:45:41.371ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
19402024-07-25T05:45:41.371ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19412024-07-25T05:45:41.391ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
19422024-07-25T05:45:41.391ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19432024-07-25T05:45:41.409ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
19442024-07-25T05:45:41.409ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19452024-07-25T05:45:41.430ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
19462024-07-25T05:45:41.430ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }
19472024-07-25T05:45:41.449ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
19482024-07-25T05:45:41.449ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }
19492024-07-25T05:45:41.469ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
19502024-07-25T05:45:41.469ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }
19512024-07-25T05:45:41.489ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
19522024-07-25T05:45:41.489ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }
19532024-07-25T05:45:41.508ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
19542024-07-25T05:45:41.508ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }
19552024-07-25T05:45:41.528ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
19562024-07-25T05:45:41.528ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }
19572024-07-25T05:45:41.547ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
19582024-07-25T05:45:41.548ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }
19592024-07-25T05:45:41.569ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
19602024-07-25T05:45:41.569ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }
19612024-07-25T05:45:41.589ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
19622024-07-25T05:45:41.589ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }
19632024-07-25T05:45:41.608ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
19642024-07-25T05:45:41.608ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }
19652024-07-25T05:45:41.627ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
19662024-07-25T05:45:41.627ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }
19672024-07-25T05:45:41.648ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
19682024-07-25T05:45:41.648ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }
19692024-07-25T05:45:41.667ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
19702024-07-25T05:45:41.667ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:177
19712024-07-25T05:45:41.667ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:177
19722024-07-25T05:45:41.667ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
19732024-07-25T05:45:41.766ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
19742024-07-25T05:45:41.766ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
19752024-07-25T05:45:41.766ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
19762024-07-25T05:45:41.766ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
19772024-07-25T05:45:41.766ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
19782024-07-25T05:45:41.766ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
19792024-07-25T05:45:41.766ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
19802024-07-25T05:45:41.766ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
19812024-07-25T05:45:41.766ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
19822024-07-25T05:45:41.766ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
19832024-07-25T05:45:41.766ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
19842024-07-25T05:45:41.766ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
19852024-07-25T05:45:41.766ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
19862024-07-25T05:45:41.766ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
19872024-07-25T05:45:41.766ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
19882024-07-25T05:45:41.766ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
19892024-07-25T05:45:41.766ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
19902024-07-25T05:45:41.766ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
19912024-07-25T05:45:41.766ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
19922024-07-25T05:45:41.766ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a", TimeNotSynchronized), ("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e", TimeNotSynchronized), ("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9", TimeNotSynchronized), ("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876", TimeNotSynchronized), ("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7", TimeNotSynchronized), ("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3", TimeNotSynchronized), ("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465", TimeNotSynchronized), ("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c", TimeNotSynchronized), ("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa", TimeNotSynchronized), ("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480", TimeNotSynchronized), ("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79", TimeNotSynchronized), ("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095", TimeNotSynchronized), ("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c", TimeNotSynchronized), ("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2", TimeNotSynchronized), ("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc", TimeNotSynchronized), ("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046", TimeNotSynchronized), ("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d", TimeNotSynchronized), ("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b", TimeNotSynchronized), ("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 497265 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:47495 req_id = 075b6451-20b8-46d9-8400-ac134b045446 response_code = 500 uri = /omicron-zones
19932024-07-25T05:45:41.767ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "075b6451-20b8-46d9-8400-ac134b045446", "content-length": "124", "date": "Thu, 25 Jul 2024 05:45:41 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "075b6451-20b8-46d9-8400-ac134b045446" } file = sled-agent/src/rack_setup/service.rs:472 retry_after = 2.208510459s sled_address = [fd00:1122:3344:101::1]:12345
19942024-07-25T05:45:43.982ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6389ba5c-9d50-4fe4-b1a2-8be3c6861cb0, kind: Service { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c }, name: Name("ntp-3f311cd5-ee38-4707-ab8a-9331491f8e7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 235])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 4f6a29ee-ca23-4fa8-be69-497060121028, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: b04db43b-dc24-43ce-9d97-32193f88e046, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 2c3f8453-233f-428f-93cc-0bfefb84929d, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: e4efbc31-e513-4290-82f5-c51d8afea0c7, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 221c5e7c-0832-4890-8b0a-f9ba08725876, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4b4998f4-10de-49f2-9326-b482137c5fd3, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: a635cc62-7516-4fd9-b490-62d6de386095, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ac9fa851-e7e1-4295-ac79-d6076526f465, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 1cb96976-2047-44b8-a79c-9fcd74d17bfa, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: a0b9fba0-c330-402f-9bc0-a7510786c98a, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d94d9602-d610-44ce-b9fc-06ded41a1480, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb1f626-0f55-4385-87de-de59052d98d2, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External } } } }, OmicronZoneConfig { id: adff3617-5320-4e42-8b10-1430a1eb0afc, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:434 sled_address = [fd00:1122:3344:101::1]:12345
19952024-07-25T05:45:43.983ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
19962024-07-25T05:45:43.983ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }
19972024-07-25T05:45:44.005ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
19982024-07-25T05:45:44.005ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }
19992024-07-25T05:45:44.028ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
20002024-07-25T05:45:44.028ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }
20012024-07-25T05:45:44.051ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
20022024-07-25T05:45:44.051ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20032024-07-25T05:45:44.071ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
20042024-07-25T05:45:44.071ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20052024-07-25T05:45:44.093ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
20062024-07-25T05:45:44.093ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20072024-07-25T05:45:44.114ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
20082024-07-25T05:45:44.114ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20092024-07-25T05:45:44.133ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
20102024-07-25T05:45:44.133ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20112024-07-25T05:45:44.154ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
20122024-07-25T05:45:44.154ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }
20132024-07-25T05:45:44.175ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
20142024-07-25T05:45:44.175ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }
20152024-07-25T05:45:44.195ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
20162024-07-25T05:45:44.195ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }
20172024-07-25T05:45:44.215ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
20182024-07-25T05:45:44.215ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }
20192024-07-25T05:45:44.236ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
20202024-07-25T05:45:44.236ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }
20212024-07-25T05:45:44.256ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
20222024-07-25T05:45:44.256ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }
20232024-07-25T05:45:44.277ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
20242024-07-25T05:45:44.277ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }
20252024-07-25T05:45:44.297ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
20262024-07-25T05:45:44.297ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }
20272024-07-25T05:45:44.315ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
20282024-07-25T05:45:44.315ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }
20292024-07-25T05:45:44.333ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
20302024-07-25T05:45:44.334ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }
20312024-07-25T05:45:44.354ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
20322024-07-25T05:45:44.354ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }
20332024-07-25T05:45:44.374ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
20342024-07-25T05:45:44.374ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }
20352024-07-25T05:45:44.393ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
20362024-07-25T05:45:44.393ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:177
20372024-07-25T05:45:44.393ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:177
20382024-07-25T05:45:44.393ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
20392024-07-25T05:45:44.491ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
20402024-07-25T05:45:44.491ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
20412024-07-25T05:45:44.491ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
20422024-07-25T05:45:44.491ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
20432024-07-25T05:45:44.491ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
20442024-07-25T05:45:44.491ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
20452024-07-25T05:45:44.491ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
20462024-07-25T05:45:44.491ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
20472024-07-25T05:45:44.491ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
20482024-07-25T05:45:44.491ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
20492024-07-25T05:45:44.491ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
20502024-07-25T05:45:44.491ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
20512024-07-25T05:45:44.491ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
20522024-07-25T05:45:44.491ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
20532024-07-25T05:45:44.491ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
20542024-07-25T05:45:44.491ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
20552024-07-25T05:45:44.491ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
20562024-07-25T05:45:44.491ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
20572024-07-25T05:45:44.491ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
20582024-07-25T05:45:44.491ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9", TimeNotSynchronized), ("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c", TimeNotSynchronized), ("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465", TimeNotSynchronized), ("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b", TimeNotSynchronized), ("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc", TimeNotSynchronized), ("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2", TimeNotSynchronized), ("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7", TimeNotSynchronized), ("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862", TimeNotSynchronized), ("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a", TimeNotSynchronized), ("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095", TimeNotSynchronized), ("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3", TimeNotSynchronized), ("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79", TimeNotSynchronized), ("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c", TimeNotSynchronized), ("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046", TimeNotSynchronized), ("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d", TimeNotSynchronized), ("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876", TimeNotSynchronized), ("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa", TimeNotSynchronized), ("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480", TimeNotSynchronized), ("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 508275 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:47495 req_id = 2da27270-e47f-4dcb-9ec1-aa04995abffe response_code = 500 uri = /omicron-zones
20592024-07-25T05:45:44.492ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "2da27270-e47f-4dcb-9ec1-aa04995abffe", "content-length": "124", "date": "Thu, 25 Jul 2024 05:45:44 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "2da27270-e47f-4dcb-9ec1-aa04995abffe" } file = sled-agent/src/rack_setup/service.rs:472 retry_after = 3.3310263s sled_address = [fd00:1122:3344:101::1]:12345
20602024-07-25T05:45:47.833ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6389ba5c-9d50-4fe4-b1a2-8be3c6861cb0, kind: Service { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c }, name: Name("ntp-3f311cd5-ee38-4707-ab8a-9331491f8e7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 235])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 4f6a29ee-ca23-4fa8-be69-497060121028, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: b04db43b-dc24-43ce-9d97-32193f88e046, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 2c3f8453-233f-428f-93cc-0bfefb84929d, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: e4efbc31-e513-4290-82f5-c51d8afea0c7, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 221c5e7c-0832-4890-8b0a-f9ba08725876, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4b4998f4-10de-49f2-9326-b482137c5fd3, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: a635cc62-7516-4fd9-b490-62d6de386095, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ac9fa851-e7e1-4295-ac79-d6076526f465, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 1cb96976-2047-44b8-a79c-9fcd74d17bfa, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: a0b9fba0-c330-402f-9bc0-a7510786c98a, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d94d9602-d610-44ce-b9fc-06ded41a1480, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb1f626-0f55-4385-87de-de59052d98d2, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External } } } }, OmicronZoneConfig { id: adff3617-5320-4e42-8b10-1430a1eb0afc, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:434 sled_address = [fd00:1122:3344:101::1]:12345
20612024-07-25T05:45:47.833ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
20622024-07-25T05:45:47.833ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }
20632024-07-25T05:45:47.857ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
20642024-07-25T05:45:47.857ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }
20652024-07-25T05:45:47.876ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
20662024-07-25T05:45:47.876ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }
20672024-07-25T05:45:47.895ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
20682024-07-25T05:45:47.895ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20692024-07-25T05:45:47.915ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
20702024-07-25T05:45:47.915ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20712024-07-25T05:45:47.936ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
20722024-07-25T05:45:47.936ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20732024-07-25T05:45:47.957ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
20742024-07-25T05:45:47.957ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20752024-07-25T05:45:47.978ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
20762024-07-25T05:45:47.978ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20772024-07-25T05:45:47.998ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
20782024-07-25T05:45:47.998ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }
20792024-07-25T05:45:48.018ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
20802024-07-25T05:45:48.018ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }
20812024-07-25T05:45:48.037ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
20822024-07-25T05:45:48.037ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }
20832024-07-25T05:45:48.057ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
20842024-07-25T05:45:48.057ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }
20852024-07-25T05:45:48.075ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
20862024-07-25T05:45:48.076ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }
20872024-07-25T05:45:48.097ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
20882024-07-25T05:45:48.097ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }
20892024-07-25T05:45:48.117ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
20902024-07-25T05:45:48.117ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }
20912024-07-25T05:45:48.137ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
20922024-07-25T05:45:48.137ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }
20932024-07-25T05:45:48.156ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
20942024-07-25T05:45:48.156ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }
20952024-07-25T05:45:48.176ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
20962024-07-25T05:45:48.176ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }
20972024-07-25T05:45:48.195ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
20982024-07-25T05:45:48.195ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }
20992024-07-25T05:45:48.215ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
21002024-07-25T05:45:48.215ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }
21012024-07-25T05:45:48.236ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
21022024-07-25T05:45:48.236ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:177
21032024-07-25T05:45:48.236ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:177
21042024-07-25T05:45:48.236ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
21052024-07-25T05:45:48.337ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
21062024-07-25T05:45:48.337ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
21072024-07-25T05:45:48.337ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
21082024-07-25T05:45:48.337ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
21092024-07-25T05:45:48.337ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
21102024-07-25T05:45:48.337ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
21112024-07-25T05:45:48.337ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
21122024-07-25T05:45:48.337ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
21132024-07-25T05:45:48.337ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
21142024-07-25T05:45:48.337ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
21152024-07-25T05:45:48.337ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
21162024-07-25T05:45:48.337ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
21172024-07-25T05:45:48.337ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
21182024-07-25T05:45:48.337ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
21192024-07-25T05:45:48.337ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
21202024-07-25T05:45:48.337ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
21212024-07-25T05:45:48.337ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
21222024-07-25T05:45:48.337ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
21232024-07-25T05:45:48.337ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
21242024-07-25T05:45:48.337ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b", TimeNotSynchronized), ("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3", TimeNotSynchronized), ("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095", TimeNotSynchronized), ("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465", TimeNotSynchronized), ("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa", TimeNotSynchronized), ("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a", TimeNotSynchronized), ("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c", TimeNotSynchronized), ("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480", TimeNotSynchronized), ("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d", TimeNotSynchronized), ("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046", TimeNotSynchronized), ("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7", TimeNotSynchronized), ("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876", TimeNotSynchronized), ("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862", TimeNotSynchronized), ("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c", TimeNotSynchronized), ("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79", TimeNotSynchronized), ("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e", TimeNotSynchronized), ("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2", TimeNotSynchronized), ("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc", TimeNotSynchronized), ("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 502552 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:47495 req_id = c62904cb-95a8-4c83-8234-e6aead661d22 response_code = 500 uri = /omicron-zones
21252024-07-25T05:45:48.338ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "c62904cb-95a8-4c83-8234-e6aead661d22", "content-length": "124", "date": "Thu, 25 Jul 2024 05:45:48 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "c62904cb-95a8-4c83-8234-e6aead661d22" } file = sled-agent/src/rack_setup/service.rs:472 retry_after = 4.301329722s sled_address = [fd00:1122:3344:101::1]:12345
21262024-07-25T05:45:52.045ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:44104
21272024-07-25T05:45:52.045ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 29 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:44104 req_id = aa92d8bc-e2fe-4d99-9383-5c41d3532dd7 response_code = 200 uri = /baseboard
21282024-07-25T05:45:52.653ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6389ba5c-9d50-4fe4-b1a2-8be3c6861cb0, kind: Service { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c }, name: Name("ntp-3f311cd5-ee38-4707-ab8a-9331491f8e7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 235])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 4f6a29ee-ca23-4fa8-be69-497060121028, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: b04db43b-dc24-43ce-9d97-32193f88e046, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 2c3f8453-233f-428f-93cc-0bfefb84929d, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: e4efbc31-e513-4290-82f5-c51d8afea0c7, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 221c5e7c-0832-4890-8b0a-f9ba08725876, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4b4998f4-10de-49f2-9326-b482137c5fd3, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: a635cc62-7516-4fd9-b490-62d6de386095, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ac9fa851-e7e1-4295-ac79-d6076526f465, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 1cb96976-2047-44b8-a79c-9fcd74d17bfa, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: a0b9fba0-c330-402f-9bc0-a7510786c98a, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d94d9602-d610-44ce-b9fc-06ded41a1480, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb1f626-0f55-4385-87de-de59052d98d2, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External } } } }, OmicronZoneConfig { id: adff3617-5320-4e42-8b10-1430a1eb0afc, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:434 sled_address = [fd00:1122:3344:101::1]:12345
21292024-07-25T05:45:52.653ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
21302024-07-25T05:45:52.653ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }
21312024-07-25T05:45:52.675ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
21322024-07-25T05:45:52.678ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }
21332024-07-25T05:45:52.695ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
21342024-07-25T05:45:52.695ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }
21352024-07-25T05:45:52.718ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
21362024-07-25T05:45:52.720ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }
21372024-07-25T05:45:52.740ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
21382024-07-25T05:45:52.740ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
21392024-07-25T05:45:52.760ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
21402024-07-25T05:45:52.760ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
21412024-07-25T05:45:52.780ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
21422024-07-25T05:45:52.780ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
21432024-07-25T05:45:52.800ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
21442024-07-25T05:45:52.800ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }
21452024-07-25T05:45:52.819ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
21462024-07-25T05:45:52.819ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }
21472024-07-25T05:45:52.839ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
21482024-07-25T05:45:52.839ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }
21492024-07-25T05:45:52.860ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
21502024-07-25T05:45:52.860ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }
21512024-07-25T05:45:52.880ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
21522024-07-25T05:45:52.880ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }
21532024-07-25T05:45:52.902ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
21542024-07-25T05:45:52.902ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }
21552024-07-25T05:45:52.922ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
21562024-07-25T05:45:52.923ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }
21572024-07-25T05:45:52.942ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
21582024-07-25T05:45:52.943ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }
21592024-07-25T05:45:52.963ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
21602024-07-25T05:45:52.963ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }
21612024-07-25T05:45:52.984ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
21622024-07-25T05:45:52.984ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }
21632024-07-25T05:45:53.005ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
21642024-07-25T05:45:53.005ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }
21652024-07-25T05:45:53.026ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
21662024-07-25T05:45:53.026ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }
21672024-07-25T05:45:53.048ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
21682024-07-25T05:45:53.048ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }
21692024-07-25T05:45:53.068ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
21702024-07-25T05:45:53.068ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:177
21712024-07-25T05:45:53.069ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:177
21722024-07-25T05:45:53.069ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
21732024-07-25T05:45:53.167ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
21742024-07-25T05:45:53.167ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
21752024-07-25T05:45:53.168ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
21762024-07-25T05:45:53.168ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
21772024-07-25T05:45:53.168ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
21782024-07-25T05:45:53.168ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
21792024-07-25T05:45:53.168ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
21802024-07-25T05:45:53.168ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
21812024-07-25T05:45:53.168ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
21822024-07-25T05:45:53.168ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
21832024-07-25T05:45:53.168ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
21842024-07-25T05:45:53.168ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
21852024-07-25T05:45:53.168ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
21862024-07-25T05:45:53.168ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
21872024-07-25T05:45:53.168ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
21882024-07-25T05:45:53.168ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
21892024-07-25T05:45:53.168ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
21902024-07-25T05:45:53.168ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
21912024-07-25T05:45:53.168ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
21922024-07-25T05:45:53.168ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa", TimeNotSynchronized), ("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9", TimeNotSynchronized), ("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095", TimeNotSynchronized), ("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79", TimeNotSynchronized), ("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d", TimeNotSynchronized), ("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046", TimeNotSynchronized), ("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7", TimeNotSynchronized), ("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465", TimeNotSynchronized), ("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e", TimeNotSynchronized), ("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c", TimeNotSynchronized), ("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c", TimeNotSynchronized), ("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc", TimeNotSynchronized), ("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b", TimeNotSynchronized), ("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876", TimeNotSynchronized), ("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a", TimeNotSynchronized), ("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480", TimeNotSynchronized), ("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3", TimeNotSynchronized), ("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2", TimeNotSynchronized), ("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 513447 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:47495 req_id = 05221f7a-196d-4d62-8f39-24c8bd9d6670 response_code = 500 uri = /omicron-zones
21932024-07-25T05:45:53.168ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "05221f7a-196d-4d62-8f39-24c8bd9d6670", "content-length": "124", "date": "Thu, 25 Jul 2024 05:45:53 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "05221f7a-196d-4d62-8f39-24c8bd9d6670" } file = sled-agent/src/rack_setup/service.rs:472 retry_after = 4.205581186s sled_address = [fd00:1122:3344:101::1]:12345
21942024-07-25T05:45:53.899ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 2.014013351s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
21952024-07-25T05:45:57.386ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6389ba5c-9d50-4fe4-b1a2-8be3c6861cb0, kind: Service { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c }, name: Name("ntp-3f311cd5-ee38-4707-ab8a-9331491f8e7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 235])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 4f6a29ee-ca23-4fa8-be69-497060121028, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: b04db43b-dc24-43ce-9d97-32193f88e046, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 2c3f8453-233f-428f-93cc-0bfefb84929d, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: e4efbc31-e513-4290-82f5-c51d8afea0c7, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 221c5e7c-0832-4890-8b0a-f9ba08725876, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4b4998f4-10de-49f2-9326-b482137c5fd3, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: a635cc62-7516-4fd9-b490-62d6de386095, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ac9fa851-e7e1-4295-ac79-d6076526f465, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 1cb96976-2047-44b8-a79c-9fcd74d17bfa, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: a0b9fba0-c330-402f-9bc0-a7510786c98a, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d94d9602-d610-44ce-b9fc-06ded41a1480, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb1f626-0f55-4385-87de-de59052d98d2, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External } } } }, OmicronZoneConfig { id: adff3617-5320-4e42-8b10-1430a1eb0afc, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:434 sled_address = [fd00:1122:3344:101::1]:12345
21962024-07-25T05:45:57.387ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
21972024-07-25T05:45:57.387ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }
21982024-07-25T05:45:57.411ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
21992024-07-25T05:45:57.411ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }
22002024-07-25T05:45:57.433ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
22012024-07-25T05:45:57.436ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }
22022024-07-25T05:45:57.454ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
22032024-07-25T05:45:57.454ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22042024-07-25T05:45:57.472ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
22052024-07-25T05:45:57.472ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22062024-07-25T05:45:57.491ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
22072024-07-25T05:45:57.491ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22082024-07-25T05:45:57.511ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
22092024-07-25T05:45:57.513ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22102024-07-25T05:45:57.531ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
22112024-07-25T05:45:57.531ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22122024-07-25T05:45:57.551ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
22132024-07-25T05:45:57.554ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }
22142024-07-25T05:45:57.572ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
22152024-07-25T05:45:57.572ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }
22162024-07-25T05:45:57.592ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
22172024-07-25T05:45:57.592ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }
22182024-07-25T05:45:57.612ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
22192024-07-25T05:45:57.612ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }
22202024-07-25T05:45:57.634ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
22212024-07-25T05:45:57.634ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }
22222024-07-25T05:45:57.655ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
22232024-07-25T05:45:57.655ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }
22242024-07-25T05:45:57.676ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
22252024-07-25T05:45:57.676ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }
22262024-07-25T05:45:57.696ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
22272024-07-25T05:45:57.696ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }
22282024-07-25T05:45:57.716ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
22292024-07-25T05:45:57.716ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }
22302024-07-25T05:45:57.737ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
22312024-07-25T05:45:57.737ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }
22322024-07-25T05:45:57.756ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
22332024-07-25T05:45:57.756ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }
22342024-07-25T05:45:57.777ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
22352024-07-25T05:45:57.778ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }
22362024-07-25T05:45:57.800ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
22372024-07-25T05:45:57.800ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:177
22382024-07-25T05:45:57.800ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:177
22392024-07-25T05:45:57.800ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
22402024-07-25T05:45:57.897ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
22412024-07-25T05:45:57.897ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
22422024-07-25T05:45:57.897ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
22432024-07-25T05:45:57.897ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
22442024-07-25T05:45:57.897ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
22452024-07-25T05:45:57.897ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
22462024-07-25T05:45:57.897ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
22472024-07-25T05:45:57.897ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
22482024-07-25T05:45:57.897ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
22492024-07-25T05:45:57.897ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
22502024-07-25T05:45:57.897ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
22512024-07-25T05:45:57.897ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
22522024-07-25T05:45:57.897ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
22532024-07-25T05:45:57.897ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
22542024-07-25T05:45:57.897ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
22552024-07-25T05:45:57.897ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
22562024-07-25T05:45:57.897ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
22572024-07-25T05:45:57.897ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
22582024-07-25T05:45:57.897ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
22592024-07-25T05:45:57.897ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095", TimeNotSynchronized), ("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc", TimeNotSynchronized), ("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79", TimeNotSynchronized), ("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7", TimeNotSynchronized), ("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b", TimeNotSynchronized), ("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa", TimeNotSynchronized), ("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c", TimeNotSynchronized), ("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480", TimeNotSynchronized), ("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465", TimeNotSynchronized), ("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862", TimeNotSynchronized), ("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e", TimeNotSynchronized), ("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2", TimeNotSynchronized), ("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9", TimeNotSynchronized), ("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d", TimeNotSynchronized), ("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876", TimeNotSynchronized), ("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046", TimeNotSynchronized), ("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3", TimeNotSynchronized), ("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a", TimeNotSynchronized), ("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 510046 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:47495 req_id = bb4662a5-720b-48d4-a51c-a90573724811 response_code = 500 uri = /omicron-zones
22602024-07-25T05:45:57.898ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "bb4662a5-720b-48d4-a51c-a90573724811", "content-length": "124", "date": "Thu, 25 Jul 2024 05:45:57 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "bb4662a5-720b-48d4-a51c-a90573724811" } file = sled-agent/src/rack_setup/service.rs:472 retry_after = 5.815637744s sled_address = [fd00:1122:3344:101::1]:12345
22612024-07-25T05:46:03.730ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6389ba5c-9d50-4fe4-b1a2-8be3c6861cb0, kind: Service { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c }, name: Name("ntp-3f311cd5-ee38-4707-ab8a-9331491f8e7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 235])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 4f6a29ee-ca23-4fa8-be69-497060121028, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: b04db43b-dc24-43ce-9d97-32193f88e046, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 2c3f8453-233f-428f-93cc-0bfefb84929d, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: e4efbc31-e513-4290-82f5-c51d8afea0c7, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 221c5e7c-0832-4890-8b0a-f9ba08725876, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4b4998f4-10de-49f2-9326-b482137c5fd3, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: a635cc62-7516-4fd9-b490-62d6de386095, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ac9fa851-e7e1-4295-ac79-d6076526f465, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 1cb96976-2047-44b8-a79c-9fcd74d17bfa, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: a0b9fba0-c330-402f-9bc0-a7510786c98a, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d94d9602-d610-44ce-b9fc-06ded41a1480, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb1f626-0f55-4385-87de-de59052d98d2, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External } } } }, OmicronZoneConfig { id: adff3617-5320-4e42-8b10-1430a1eb0afc, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:434 sled_address = [fd00:1122:3344:101::1]:12345
22622024-07-25T05:46:03.731ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
22632024-07-25T05:46:03.731ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }
22642024-07-25T05:46:03.755ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
22652024-07-25T05:46:03.755ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }
22662024-07-25T05:46:03.777ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
22672024-07-25T05:46:03.777ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }
22682024-07-25T05:46:03.798ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
22692024-07-25T05:46:03.798ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22702024-07-25T05:46:03.818ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
22712024-07-25T05:46:03.818ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22722024-07-25T05:46:03.838ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
22732024-07-25T05:46:03.838ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22742024-07-25T05:46:03.858ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
22752024-07-25T05:46:03.859ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22762024-07-25T05:46:03.879ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
22772024-07-25T05:46:03.879ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22782024-07-25T05:46:03.898ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
22792024-07-25T05:46:03.899ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }
22802024-07-25T05:46:03.923ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
22812024-07-25T05:46:03.923ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }
22822024-07-25T05:46:03.944ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
22832024-07-25T05:46:03.944ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }
22842024-07-25T05:46:03.966ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
22852024-07-25T05:46:03.966ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }
22862024-07-25T05:46:03.987ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
22872024-07-25T05:46:03.987ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }
22882024-07-25T05:46:04.008ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
22892024-07-25T05:46:04.008ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }
22902024-07-25T05:46:04.029ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
22912024-07-25T05:46:04.029ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }
22922024-07-25T05:46:04.050ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
22932024-07-25T05:46:04.050ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }
22942024-07-25T05:46:04.071ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
22952024-07-25T05:46:04.071ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }
22962024-07-25T05:46:04.091ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
22972024-07-25T05:46:04.091ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }
22982024-07-25T05:46:04.112ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
22992024-07-25T05:46:04.112ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }
23002024-07-25T05:46:04.133ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
23012024-07-25T05:46:04.133ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }
23022024-07-25T05:46:04.153ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
23032024-07-25T05:46:04.153ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:177
23042024-07-25T05:46:04.153ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:177
23052024-07-25T05:46:04.154ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
23062024-07-25T05:46:04.256ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
23072024-07-25T05:46:04.256ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
23082024-07-25T05:46:04.256ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
23092024-07-25T05:46:04.256ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
23102024-07-25T05:46:04.256ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
23112024-07-25T05:46:04.256ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
23122024-07-25T05:46:04.256ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
23132024-07-25T05:46:04.256ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
23142024-07-25T05:46:04.256ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
23152024-07-25T05:46:04.256ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
23162024-07-25T05:46:04.256ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
23172024-07-25T05:46:04.256ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
23182024-07-25T05:46:04.256ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
23192024-07-25T05:46:04.256ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
23202024-07-25T05:46:04.256ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
23212024-07-25T05:46:04.256ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
23222024-07-25T05:46:04.256ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
23232024-07-25T05:46:04.256ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
23242024-07-25T05:46:04.256ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
23252024-07-25T05:46:04.257ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7", TimeNotSynchronized), ("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876", TimeNotSynchronized), ("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465", TimeNotSynchronized), ("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2", TimeNotSynchronized), ("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046", TimeNotSynchronized), ("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3", TimeNotSynchronized), ("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c", TimeNotSynchronized), ("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a", TimeNotSynchronized), ("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa", TimeNotSynchronized), ("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9", TimeNotSynchronized), ("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095", TimeNotSynchronized), ("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc", TimeNotSynchronized), ("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d", TimeNotSynchronized), ("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862", TimeNotSynchronized), ("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e", TimeNotSynchronized), ("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b", TimeNotSynchronized), ("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480", TimeNotSynchronized), ("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79", TimeNotSynchronized), ("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 524813 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:47495 req_id = f5e81b75-df7e-418c-abc1-94882621c903 response_code = 500 uri = /omicron-zones
23262024-07-25T05:46:04.257ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "f5e81b75-df7e-418c-abc1-94882621c903", "content-length": "124", "date": "Thu, 25 Jul 2024 05:46:04 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "f5e81b75-df7e-418c-abc1-94882621c903" } file = sled-agent/src/rack_setup/service.rs:472 retry_after = 4.059541631s sled_address = [fd00:1122:3344:101::1]:12345
23272024-07-25T05:46:08.329ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6389ba5c-9d50-4fe4-b1a2-8be3c6861cb0, kind: Service { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c }, name: Name("ntp-3f311cd5-ee38-4707-ab8a-9331491f8e7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 235])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 4f6a29ee-ca23-4fa8-be69-497060121028, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: b04db43b-dc24-43ce-9d97-32193f88e046, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 2c3f8453-233f-428f-93cc-0bfefb84929d, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: e4efbc31-e513-4290-82f5-c51d8afea0c7, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 221c5e7c-0832-4890-8b0a-f9ba08725876, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4b4998f4-10de-49f2-9326-b482137c5fd3, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: a635cc62-7516-4fd9-b490-62d6de386095, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ac9fa851-e7e1-4295-ac79-d6076526f465, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 1cb96976-2047-44b8-a79c-9fcd74d17bfa, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: a0b9fba0-c330-402f-9bc0-a7510786c98a, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d94d9602-d610-44ce-b9fc-06ded41a1480, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb1f626-0f55-4385-87de-de59052d98d2, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External } } } }, OmicronZoneConfig { id: adff3617-5320-4e42-8b10-1430a1eb0afc, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:434 sled_address = [fd00:1122:3344:101::1]:12345
23282024-07-25T05:46:08.329ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
23292024-07-25T05:46:08.329ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }
23302024-07-25T05:46:08.354ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
23312024-07-25T05:46:08.354ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }
23322024-07-25T05:46:08.376ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
23332024-07-25T05:46:08.376ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }
23342024-07-25T05:46:08.398ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
23352024-07-25T05:46:08.398ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }
23362024-07-25T05:46:08.419ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
23372024-07-25T05:46:08.419ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
23382024-07-25T05:46:08.441ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
23392024-07-25T05:46:08.442ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
23402024-07-25T05:46:08.462ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
23412024-07-25T05:46:08.462ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
23422024-07-25T05:46:08.483ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
23432024-07-25T05:46:08.483ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }
23442024-07-25T05:46:08.504ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
23452024-07-25T05:46:08.504ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }
23462024-07-25T05:46:08.525ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
23472024-07-25T05:46:08.525ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }
23482024-07-25T05:46:08.545ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
23492024-07-25T05:46:08.546ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }
23502024-07-25T05:46:08.565ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
23512024-07-25T05:46:08.565ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }
23522024-07-25T05:46:08.586ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
23532024-07-25T05:46:08.586ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }
23542024-07-25T05:46:08.608ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
23552024-07-25T05:46:08.608ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }
23562024-07-25T05:46:08.632ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
23572024-07-25T05:46:08.632ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }
23582024-07-25T05:46:08.656ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
23592024-07-25T05:46:08.656ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }
23602024-07-25T05:46:08.679ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
23612024-07-25T05:46:08.679ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }
23622024-07-25T05:46:08.700ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
23632024-07-25T05:46:08.700ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }
23642024-07-25T05:46:08.722ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
23652024-07-25T05:46:08.722ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }
23662024-07-25T05:46:08.744ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
23672024-07-25T05:46:08.744ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }
23682024-07-25T05:46:08.765ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
23692024-07-25T05:46:08.765ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:177
23702024-07-25T05:46:08.765ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:177
23712024-07-25T05:46:08.765ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
23722024-07-25T05:46:08.867ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
23732024-07-25T05:46:08.867ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
23742024-07-25T05:46:08.867ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
23752024-07-25T05:46:08.867ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
23762024-07-25T05:46:08.867ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
23772024-07-25T05:46:08.867ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
23782024-07-25T05:46:08.867ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
23792024-07-25T05:46:08.867ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
23802024-07-25T05:46:08.867ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
23812024-07-25T05:46:08.867ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
23822024-07-25T05:46:08.867ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
23832024-07-25T05:46:08.867ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
23842024-07-25T05:46:08.867ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
23852024-07-25T05:46:08.867ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
23862024-07-25T05:46:08.867ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
23872024-07-25T05:46:08.867ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
23882024-07-25T05:46:08.867ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
23892024-07-25T05:46:08.867ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
23902024-07-25T05:46:08.867ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
23912024-07-25T05:46:08.867ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c", TimeNotSynchronized), ("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046", TimeNotSynchronized), ("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2", TimeNotSynchronized), ("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862", TimeNotSynchronized), ("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc", TimeNotSynchronized), ("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9", TimeNotSynchronized), ("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79", TimeNotSynchronized), ("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3", TimeNotSynchronized), ("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c", TimeNotSynchronized), ("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d", TimeNotSynchronized), ("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b", TimeNotSynchronized), ("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465", TimeNotSynchronized), ("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa", TimeNotSynchronized), ("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7", TimeNotSynchronized), ("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876", TimeNotSynchronized), ("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095", TimeNotSynchronized), ("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a", TimeNotSynchronized), ("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480", TimeNotSynchronized), ("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 536449 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:47495 req_id = dd86c9dd-7184-4622-9a76-979ea913ab6e response_code = 500 uri = /omicron-zones
23922024-07-25T05:46:08.867ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "dd86c9dd-7184-4622-9a76-979ea913ab6e", "content-length": "124", "date": "Thu, 25 Jul 2024 05:46:08 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "dd86c9dd-7184-4622-9a76-979ea913ab6e" } file = sled-agent/src/rack_setup/service.rs:472 retry_after = 8.248718398s sled_address = [fd00:1122:3344:101::1]:12345
23932024-07-25T05:46:10.995ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 6.112851105s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
23942024-07-25T05:46:17.155ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6389ba5c-9d50-4fe4-b1a2-8be3c6861cb0, kind: Service { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c }, name: Name("ntp-3f311cd5-ee38-4707-ab8a-9331491f8e7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 235])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 4f6a29ee-ca23-4fa8-be69-497060121028, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: b04db43b-dc24-43ce-9d97-32193f88e046, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 2c3f8453-233f-428f-93cc-0bfefb84929d, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: e4efbc31-e513-4290-82f5-c51d8afea0c7, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 221c5e7c-0832-4890-8b0a-f9ba08725876, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4b4998f4-10de-49f2-9326-b482137c5fd3, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: a635cc62-7516-4fd9-b490-62d6de386095, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ac9fa851-e7e1-4295-ac79-d6076526f465, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 1cb96976-2047-44b8-a79c-9fcd74d17bfa, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: a0b9fba0-c330-402f-9bc0-a7510786c98a, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d94d9602-d610-44ce-b9fc-06ded41a1480, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb1f626-0f55-4385-87de-de59052d98d2, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External } } } }, OmicronZoneConfig { id: adff3617-5320-4e42-8b10-1430a1eb0afc, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:434 sled_address = [fd00:1122:3344:101::1]:12345
23952024-07-25T05:46:17.156ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
23962024-07-25T05:46:17.156ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }
23972024-07-25T05:46:17.179ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
23982024-07-25T05:46:17.179ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }
23992024-07-25T05:46:17.199ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
24002024-07-25T05:46:17.199ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }
24012024-07-25T05:46:17.218ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
24022024-07-25T05:46:17.218ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24032024-07-25T05:46:17.237ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
24042024-07-25T05:46:17.237ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24052024-07-25T05:46:17.256ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
24062024-07-25T05:46:17.256ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24072024-07-25T05:46:17.274ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
24082024-07-25T05:46:17.277ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24092024-07-25T05:46:17.292ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
24102024-07-25T05:46:17.292ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24112024-07-25T05:46:17.310ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
24122024-07-25T05:46:17.311ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }
24132024-07-25T05:46:17.329ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
24142024-07-25T05:46:17.329ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }
24152024-07-25T05:46:17.347ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
24162024-07-25T05:46:17.347ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }
24172024-07-25T05:46:17.366ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
24182024-07-25T05:46:17.366ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }
24192024-07-25T05:46:17.385ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
24202024-07-25T05:46:17.385ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }
24212024-07-25T05:46:17.403ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
24222024-07-25T05:46:17.403ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }
24232024-07-25T05:46:17.421ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
24242024-07-25T05:46:17.421ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }
24252024-07-25T05:46:17.440ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
24262024-07-25T05:46:17.440ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }
24272024-07-25T05:46:17.458ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
24282024-07-25T05:46:17.458ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }
24292024-07-25T05:46:17.477ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
24302024-07-25T05:46:17.477ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }
24312024-07-25T05:46:17.496ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
24322024-07-25T05:46:17.496ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }
24332024-07-25T05:46:17.514ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
24342024-07-25T05:46:17.514ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }
24352024-07-25T05:46:17.532ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
24362024-07-25T05:46:17.533ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:177
24372024-07-25T05:46:17.533ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:177
24382024-07-25T05:46:17.533ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
24392024-07-25T05:46:17.628ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
24402024-07-25T05:46:17.628ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
24412024-07-25T05:46:17.628ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
24422024-07-25T05:46:17.628ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
24432024-07-25T05:46:17.628ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
24442024-07-25T05:46:17.628ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
24452024-07-25T05:46:17.628ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
24462024-07-25T05:46:17.628ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
24472024-07-25T05:46:17.628ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
24482024-07-25T05:46:17.628ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
24492024-07-25T05:46:17.628ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
24502024-07-25T05:46:17.628ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
24512024-07-25T05:46:17.628ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
24522024-07-25T05:46:17.628ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
24532024-07-25T05:46:17.628ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
24542024-07-25T05:46:17.628ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
24552024-07-25T05:46:17.628ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
24562024-07-25T05:46:17.628ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
24572024-07-25T05:46:17.628ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
24582024-07-25T05:46:17.628ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d", TimeNotSynchronized), ("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c", TimeNotSynchronized), ("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2", TimeNotSynchronized), ("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9", TimeNotSynchronized), ("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b", TimeNotSynchronized), ("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a", TimeNotSynchronized), ("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046", TimeNotSynchronized), ("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c", TimeNotSynchronized), ("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79", TimeNotSynchronized), ("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3", TimeNotSynchronized), ("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862", TimeNotSynchronized), ("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa", TimeNotSynchronized), ("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc", TimeNotSynchronized), ("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465", TimeNotSynchronized), ("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7", TimeNotSynchronized), ("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e", TimeNotSynchronized), ("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095", TimeNotSynchronized), ("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876", TimeNotSynchronized), ("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 472094 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:47495 req_id = 2e6446d6-f287-46f7-88cc-04f20b8283a5 response_code = 500 uri = /omicron-zones
24592024-07-25T05:46:17.629ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "2e6446d6-f287-46f7-88cc-04f20b8283a5", "content-length": "124", "date": "Thu, 25 Jul 2024 05:46:17 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "2e6446d6-f287-46f7-88cc-04f20b8283a5" } file = sled-agent/src/rack_setup/service.rs:472 retry_after = 7.529878821s sled_address = [fd00:1122:3344:101::1]:12345
24602024-07-25T05:46:22.161ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:55774
24612024-07-25T05:46:22.162ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 817 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:55774 req_id = 8a9cc9b9-1a77-476a-87da-f4c79c77463a response_code = 200 uri = /baseboard
24622024-07-25T05:46:25.180ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6389ba5c-9d50-4fe4-b1a2-8be3c6861cb0, kind: Service { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c }, name: Name("ntp-3f311cd5-ee38-4707-ab8a-9331491f8e7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 235])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 4f6a29ee-ca23-4fa8-be69-497060121028, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: b04db43b-dc24-43ce-9d97-32193f88e046, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 2c3f8453-233f-428f-93cc-0bfefb84929d, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: e4efbc31-e513-4290-82f5-c51d8afea0c7, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 221c5e7c-0832-4890-8b0a-f9ba08725876, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4b4998f4-10de-49f2-9326-b482137c5fd3, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: a635cc62-7516-4fd9-b490-62d6de386095, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ac9fa851-e7e1-4295-ac79-d6076526f465, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 1cb96976-2047-44b8-a79c-9fcd74d17bfa, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: a0b9fba0-c330-402f-9bc0-a7510786c98a, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d94d9602-d610-44ce-b9fc-06ded41a1480, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb1f626-0f55-4385-87de-de59052d98d2, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External } } } }, OmicronZoneConfig { id: adff3617-5320-4e42-8b10-1430a1eb0afc, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:434 sled_address = [fd00:1122:3344:101::1]:12345
24632024-07-25T05:46:25.181ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
24642024-07-25T05:46:25.181ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }
24652024-07-25T05:46:25.204ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
24662024-07-25T05:46:25.204ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }
24672024-07-25T05:46:25.223ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
24682024-07-25T05:46:25.223ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }
24692024-07-25T05:46:25.242ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
24702024-07-25T05:46:25.242ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24712024-07-25T05:46:25.260ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
24722024-07-25T05:46:25.260ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24732024-07-25T05:46:25.277ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
24742024-07-25T05:46:25.278ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24752024-07-25T05:46:25.295ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
24762024-07-25T05:46:25.295ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24772024-07-25T05:46:25.314ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
24782024-07-25T05:46:25.314ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24792024-07-25T05:46:25.331ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
24802024-07-25T05:46:25.331ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }
24812024-07-25T05:46:25.351ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
24822024-07-25T05:46:25.351ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }
24832024-07-25T05:46:25.371ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
24842024-07-25T05:46:25.371ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }
24852024-07-25T05:46:25.391ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
24862024-07-25T05:46:25.391ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }
24872024-07-25T05:46:25.415ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
24882024-07-25T05:46:25.415ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }
24892024-07-25T05:46:25.435ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
24902024-07-25T05:46:25.435ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }
24912024-07-25T05:46:25.456ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
24922024-07-25T05:46:25.456ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }
24932024-07-25T05:46:25.475ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
24942024-07-25T05:46:25.475ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }
24952024-07-25T05:46:25.494ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
24962024-07-25T05:46:25.494ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }
24972024-07-25T05:46:25.513ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
24982024-07-25T05:46:25.514ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }
24992024-07-25T05:46:25.532ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
25002024-07-25T05:46:25.532ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }
25012024-07-25T05:46:25.551ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
25022024-07-25T05:46:25.551ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }
25032024-07-25T05:46:25.570ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
25042024-07-25T05:46:25.570ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:177
25052024-07-25T05:46:25.570ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:177
25062024-07-25T05:46:25.570ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
25072024-07-25T05:46:25.667ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
25082024-07-25T05:46:25.667ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
25092024-07-25T05:46:25.667ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
25102024-07-25T05:46:25.667ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
25112024-07-25T05:46:25.667ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
25122024-07-25T05:46:25.667ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
25132024-07-25T05:46:25.667ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
25142024-07-25T05:46:25.667ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
25152024-07-25T05:46:25.667ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
25162024-07-25T05:46:25.667ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
25172024-07-25T05:46:25.667ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
25182024-07-25T05:46:25.667ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
25192024-07-25T05:46:25.667ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
25202024-07-25T05:46:25.667ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
25212024-07-25T05:46:25.667ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
25222024-07-25T05:46:25.667ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
25232024-07-25T05:46:25.667ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
25242024-07-25T05:46:25.667ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
25252024-07-25T05:46:25.667ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
25262024-07-25T05:46:25.667ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c", TimeNotSynchronized), ("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480", TimeNotSynchronized), ("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79", TimeNotSynchronized), ("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095", TimeNotSynchronized), ("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc", TimeNotSynchronized), ("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862", TimeNotSynchronized), ("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e", TimeNotSynchronized), ("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9", TimeNotSynchronized), ("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3", TimeNotSynchronized), ("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876", TimeNotSynchronized), ("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b", TimeNotSynchronized), ("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa", TimeNotSynchronized), ("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a", TimeNotSynchronized), ("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2", TimeNotSynchronized), ("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7", TimeNotSynchronized), ("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c", TimeNotSynchronized), ("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046", TimeNotSynchronized), ("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d", TimeNotSynchronized), ("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 485558 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:47495 req_id = 07eb67e9-b3c7-497d-b5da-b034ee2fe4a4 response_code = 500 uri = /omicron-zones
25272024-07-25T05:46:25.668ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "07eb67e9-b3c7-497d-b5da-b034ee2fe4a4", "content-length": "124", "date": "Thu, 25 Jul 2024 05:46:25 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "07eb67e9-b3c7-497d-b5da-b034ee2fe4a4" } file = sled-agent/src/rack_setup/service.rs:472 retry_after = 6.546953853s sled_address = [fd00:1122:3344:101::1]:12345
25282024-07-25T05:46:32.198ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 21.728864219s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
25292024-07-25T05:46:32.234ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6389ba5c-9d50-4fe4-b1a2-8be3c6861cb0, kind: Service { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c }, name: Name("ntp-3f311cd5-ee38-4707-ab8a-9331491f8e7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 235])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 4f6a29ee-ca23-4fa8-be69-497060121028, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: b04db43b-dc24-43ce-9d97-32193f88e046, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 2c3f8453-233f-428f-93cc-0bfefb84929d, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: e4efbc31-e513-4290-82f5-c51d8afea0c7, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 221c5e7c-0832-4890-8b0a-f9ba08725876, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4b4998f4-10de-49f2-9326-b482137c5fd3, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: a635cc62-7516-4fd9-b490-62d6de386095, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ac9fa851-e7e1-4295-ac79-d6076526f465, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 1cb96976-2047-44b8-a79c-9fcd74d17bfa, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: a0b9fba0-c330-402f-9bc0-a7510786c98a, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d94d9602-d610-44ce-b9fc-06ded41a1480, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb1f626-0f55-4385-87de-de59052d98d2, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External } } } }, OmicronZoneConfig { id: adff3617-5320-4e42-8b10-1430a1eb0afc, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:434 sled_address = [fd00:1122:3344:101::1]:12345
25302024-07-25T05:46:32.234ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
25312024-07-25T05:46:32.234ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }
25322024-07-25T05:46:32.256ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
25332024-07-25T05:46:32.259ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }
25342024-07-25T05:46:32.276ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
25352024-07-25T05:46:32.276ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }
25362024-07-25T05:46:32.294ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
25372024-07-25T05:46:32.294ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }
25382024-07-25T05:46:32.312ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
25392024-07-25T05:46:32.312ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
25402024-07-25T05:46:32.330ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
25412024-07-25T05:46:32.330ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
25422024-07-25T05:46:32.348ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
25432024-07-25T05:46:32.348ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
25442024-07-25T05:46:32.366ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
25452024-07-25T05:46:32.366ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }
25462024-07-25T05:46:32.385ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
25472024-07-25T05:46:32.385ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }
25482024-07-25T05:46:32.403ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
25492024-07-25T05:46:32.403ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }
25502024-07-25T05:46:32.421ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
25512024-07-25T05:46:32.424ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }
25522024-07-25T05:46:32.440ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
25532024-07-25T05:46:32.440ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }
25542024-07-25T05:46:32.458ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
25552024-07-25T05:46:32.458ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }
25562024-07-25T05:46:32.478ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
25572024-07-25T05:46:32.478ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }
25582024-07-25T05:46:32.496ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
25592024-07-25T05:46:32.496ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }
25602024-07-25T05:46:32.514ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
25612024-07-25T05:46:32.514ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }
25622024-07-25T05:46:32.533ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
25632024-07-25T05:46:32.533ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }
25642024-07-25T05:46:32.551ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
25652024-07-25T05:46:32.551ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }
25662024-07-25T05:46:32.570ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
25672024-07-25T05:46:32.570ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }
25682024-07-25T05:46:32.589ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
25692024-07-25T05:46:32.589ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }
25702024-07-25T05:46:32.607ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
25712024-07-25T05:46:32.607ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:177
25722024-07-25T05:46:32.608ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:177
25732024-07-25T05:46:32.608ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
25742024-07-25T05:46:32.701ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
25752024-07-25T05:46:32.701ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
25762024-07-25T05:46:32.701ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
25772024-07-25T05:46:32.701ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
25782024-07-25T05:46:32.701ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
25792024-07-25T05:46:32.701ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
25802024-07-25T05:46:32.701ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
25812024-07-25T05:46:32.701ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
25822024-07-25T05:46:32.701ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
25832024-07-25T05:46:32.701ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
25842024-07-25T05:46:32.701ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
25852024-07-25T05:46:32.701ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
25862024-07-25T05:46:32.701ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
25872024-07-25T05:46:32.701ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
25882024-07-25T05:46:32.701ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
25892024-07-25T05:46:32.701ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
25902024-07-25T05:46:32.701ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
25912024-07-25T05:46:32.701ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
25922024-07-25T05:46:32.702ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
25932024-07-25T05:46:32.702ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a", TimeNotSynchronized), ("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862", TimeNotSynchronized), ("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095", TimeNotSynchronized), ("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e", TimeNotSynchronized), ("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3", TimeNotSynchronized), ("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465", TimeNotSynchronized), ("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa", TimeNotSynchronized), ("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480", TimeNotSynchronized), ("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79", TimeNotSynchronized), ("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7", TimeNotSynchronized), ("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b", TimeNotSynchronized), ("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c", TimeNotSynchronized), ("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2", TimeNotSynchronized), ("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc", TimeNotSynchronized), ("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9", TimeNotSynchronized), ("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d", TimeNotSynchronized), ("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046", TimeNotSynchronized), ("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876", TimeNotSynchronized), ("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 466518 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:47495 req_id = 140661b4-14dc-49a6-a219-d42e1a73fe5e response_code = 500 uri = /omicron-zones
25942024-07-25T05:46:32.702ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "140661b4-14dc-49a6-a219-d42e1a73fe5e", "content-length": "124", "date": "Thu, 25 Jul 2024 05:46:32 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "140661b4-14dc-49a6-a219-d42e1a73fe5e" } file = sled-agent/src/rack_setup/service.rs:472 retry_after = 10.494152653s sled_address = [fd00:1122:3344:101::1]:12345
25952024-07-25T05:46:43.226ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6389ba5c-9d50-4fe4-b1a2-8be3c6861cb0, kind: Service { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c }, name: Name("ntp-3f311cd5-ee38-4707-ab8a-9331491f8e7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 235])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 4f6a29ee-ca23-4fa8-be69-497060121028, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: b04db43b-dc24-43ce-9d97-32193f88e046, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 2c3f8453-233f-428f-93cc-0bfefb84929d, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: e4efbc31-e513-4290-82f5-c51d8afea0c7, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 221c5e7c-0832-4890-8b0a-f9ba08725876, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4b4998f4-10de-49f2-9326-b482137c5fd3, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: a635cc62-7516-4fd9-b490-62d6de386095, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ac9fa851-e7e1-4295-ac79-d6076526f465, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 1cb96976-2047-44b8-a79c-9fcd74d17bfa, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: a0b9fba0-c330-402f-9bc0-a7510786c98a, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d94d9602-d610-44ce-b9fc-06ded41a1480, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb1f626-0f55-4385-87de-de59052d98d2, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External } } } }, OmicronZoneConfig { id: adff3617-5320-4e42-8b10-1430a1eb0afc, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:434 sled_address = [fd00:1122:3344:101::1]:12345
25962024-07-25T05:46:43.227ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
25972024-07-25T05:46:43.227ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }
25982024-07-25T05:46:43.250ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
25992024-07-25T05:46:43.250ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }
26002024-07-25T05:46:43.270ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
26012024-07-25T05:46:43.270ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }
26022024-07-25T05:46:43.289ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
26032024-07-25T05:46:43.289ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }
26042024-07-25T05:46:43.311ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
26052024-07-25T05:46:43.311ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
26062024-07-25T05:46:43.329ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
26072024-07-25T05:46:43.329ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
26082024-07-25T05:46:43.347ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
26092024-07-25T05:46:43.347ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
26102024-07-25T05:46:43.366ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
26112024-07-25T05:46:43.366ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }
26122024-07-25T05:46:43.386ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
26132024-07-25T05:46:43.386ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }
26142024-07-25T05:46:43.406ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
26152024-07-25T05:46:43.406ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }
26162024-07-25T05:46:43.424ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
26172024-07-25T05:46:43.427ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }
26182024-07-25T05:46:43.444ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
26192024-07-25T05:46:43.444ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }
26202024-07-25T05:46:43.462ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
26212024-07-25T05:46:43.462ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }
26222024-07-25T05:46:43.481ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
26232024-07-25T05:46:43.481ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }
26242024-07-25T05:46:43.499ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
26252024-07-25T05:46:43.500ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }
26262024-07-25T05:46:43.518ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
26272024-07-25T05:46:43.518ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }
26282024-07-25T05:46:43.537ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
26292024-07-25T05:46:43.537ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }
26302024-07-25T05:46:43.555ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
26312024-07-25T05:46:43.556ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }
26322024-07-25T05:46:43.574ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
26332024-07-25T05:46:43.574ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }
26342024-07-25T05:46:43.593ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
26352024-07-25T05:46:43.593ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }
26362024-07-25T05:46:43.612ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
26372024-07-25T05:46:43.612ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:177
26382024-07-25T05:46:43.613ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:177
26392024-07-25T05:46:43.613ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
26402024-07-25T05:46:43.708ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
26412024-07-25T05:46:43.708ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
26422024-07-25T05:46:43.708ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
26432024-07-25T05:46:43.708ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
26442024-07-25T05:46:43.708ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
26452024-07-25T05:46:43.708ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
26462024-07-25T05:46:43.708ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
26472024-07-25T05:46:43.708ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
26482024-07-25T05:46:43.708ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
26492024-07-25T05:46:43.708ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
26502024-07-25T05:46:43.708ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
26512024-07-25T05:46:43.708ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
26522024-07-25T05:46:43.708ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
26532024-07-25T05:46:43.708ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
26542024-07-25T05:46:43.708ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
26552024-07-25T05:46:43.708ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
26562024-07-25T05:46:43.708ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
26572024-07-25T05:46:43.708ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
26582024-07-25T05:46:43.708ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
26592024-07-25T05:46:43.708ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046", TimeNotSynchronized), ("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b", TimeNotSynchronized), ("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3", TimeNotSynchronized), ("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa", TimeNotSynchronized), ("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a", TimeNotSynchronized), ("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c", TimeNotSynchronized), ("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480", TimeNotSynchronized), ("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c", TimeNotSynchronized), ("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e", TimeNotSynchronized), ("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2", TimeNotSynchronized), ("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9", TimeNotSynchronized), ("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d", TimeNotSynchronized), ("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7", TimeNotSynchronized), ("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862", TimeNotSynchronized), ("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465", TimeNotSynchronized), ("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc", TimeNotSynchronized), ("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79", TimeNotSynchronized), ("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876", TimeNotSynchronized), ("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 480518 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:47495 req_id = fe18f56b-9776-468e-9bc0-8b26bb2cae19 response_code = 500 uri = /omicron-zones
26602024-07-25T05:46:43.708ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "fe18f56b-9776-468e-9bc0-8b26bb2cae19", "content-length": "124", "date": "Thu, 25 Jul 2024 05:46:43 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "fe18f56b-9776-468e-9bc0-8b26bb2cae19" } file = sled-agent/src/rack_setup/service.rs:472 retry_after = 18.50844288s sled_address = [fd00:1122:3344:101::1]:12345
26612024-07-25T05:46:52.276ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:45289
26622024-07-25T05:46:52.276ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 39 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:45289 req_id = e81dcabc-cc6c-42a2-87e4-2c63d45d5cec response_code = 200 uri = /baseboard
26632024-07-25T05:47:02.269ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6389ba5c-9d50-4fe4-b1a2-8be3c6861cb0, kind: Service { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c }, name: Name("ntp-3f311cd5-ee38-4707-ab8a-9331491f8e7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 235])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 4f6a29ee-ca23-4fa8-be69-497060121028, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: b04db43b-dc24-43ce-9d97-32193f88e046, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 2c3f8453-233f-428f-93cc-0bfefb84929d, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: e4efbc31-e513-4290-82f5-c51d8afea0c7, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 221c5e7c-0832-4890-8b0a-f9ba08725876, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4b4998f4-10de-49f2-9326-b482137c5fd3, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: a635cc62-7516-4fd9-b490-62d6de386095, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ac9fa851-e7e1-4295-ac79-d6076526f465, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 1cb96976-2047-44b8-a79c-9fcd74d17bfa, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: a0b9fba0-c330-402f-9bc0-a7510786c98a, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d94d9602-d610-44ce-b9fc-06ded41a1480, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb1f626-0f55-4385-87de-de59052d98d2, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External } } } }, OmicronZoneConfig { id: adff3617-5320-4e42-8b10-1430a1eb0afc, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:434 sled_address = [fd00:1122:3344:101::1]:12345
26642024-07-25T05:47:02.269ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
26652024-07-25T05:47:02.269ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }
26662024-07-25T05:47:02.292ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
26672024-07-25T05:47:02.292ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }
26682024-07-25T05:47:02.310ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
26692024-07-25T05:47:02.310ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }
26702024-07-25T05:47:02.329ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
26712024-07-25T05:47:02.329ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }
26722024-07-25T05:47:02.348ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
26732024-07-25T05:47:02.348ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
26742024-07-25T05:47:02.366ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
26752024-07-25T05:47:02.366ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
26762024-07-25T05:47:02.383ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
26772024-07-25T05:47:02.383ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
26782024-07-25T05:47:02.402ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
26792024-07-25T05:47:02.402ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }
26802024-07-25T05:47:02.420ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
26812024-07-25T05:47:02.420ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }
26822024-07-25T05:47:02.439ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
26832024-07-25T05:47:02.439ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }
26842024-07-25T05:47:02.457ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
26852024-07-25T05:47:02.457ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }
26862024-07-25T05:47:02.476ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
26872024-07-25T05:47:02.476ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }
26882024-07-25T05:47:02.497ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
26892024-07-25T05:47:02.497ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }
26902024-07-25T05:47:02.514ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
26912024-07-25T05:47:02.514ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }
26922024-07-25T05:47:02.533ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
26932024-07-25T05:47:02.533ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }
26942024-07-25T05:47:02.551ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
26952024-07-25T05:47:02.551ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }
26962024-07-25T05:47:02.570ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
26972024-07-25T05:47:02.570ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }
26982024-07-25T05:47:02.591ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
26992024-07-25T05:47:02.591ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }
27002024-07-25T05:47:02.608ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
27012024-07-25T05:47:02.608ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }
27022024-07-25T05:47:02.627ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
27032024-07-25T05:47:02.627ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }
27042024-07-25T05:47:02.644ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
27052024-07-25T05:47:02.644ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:177
27062024-07-25T05:47:02.645ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:177
27072024-07-25T05:47:02.645ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
27082024-07-25T05:47:02.738ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
27092024-07-25T05:47:02.738ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
27102024-07-25T05:47:02.738ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
27112024-07-25T05:47:02.738ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
27122024-07-25T05:47:02.738ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
27132024-07-25T05:47:02.738ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
27142024-07-25T05:47:02.738ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
27152024-07-25T05:47:02.738ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
27162024-07-25T05:47:02.738ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
27172024-07-25T05:47:02.738ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
27182024-07-25T05:47:02.738ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
27192024-07-25T05:47:02.738ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
27202024-07-25T05:47:02.738ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
27212024-07-25T05:47:02.738ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
27222024-07-25T05:47:02.738ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
27232024-07-25T05:47:02.738ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
27242024-07-25T05:47:02.738ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
27252024-07-25T05:47:02.738ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
27262024-07-25T05:47:02.738ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
27272024-07-25T05:47:02.738ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c", TimeNotSynchronized), ("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a", TimeNotSynchronized), ("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480", TimeNotSynchronized), ("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2", TimeNotSynchronized), ("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc", TimeNotSynchronized), ("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046", TimeNotSynchronized), ("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862", TimeNotSynchronized), ("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9", TimeNotSynchronized), ("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3", TimeNotSynchronized), ("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c", TimeNotSynchronized), ("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79", TimeNotSynchronized), ("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e", TimeNotSynchronized), ("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465", TimeNotSynchronized), ("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d", TimeNotSynchronized), ("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7", TimeNotSynchronized), ("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876", TimeNotSynchronized), ("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095", TimeNotSynchronized), ("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b", TimeNotSynchronized), ("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 467963 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:47495 req_id = b979f377-2c30-48db-bce9-c83b90f10cd5 response_code = 500 uri = /omicron-zones
27282024-07-25T05:47:02.738ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "b979f377-2c30-48db-bce9-c83b90f10cd5", "content-length": "124", "date": "Thu, 25 Jul 2024 05:47:02 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "b979f377-2c30-48db-bce9-c83b90f10cd5" } file = sled-agent/src/rack_setup/service.rs:472 retry_after = 8.905614665s sled_address = [fd00:1122:3344:101::1]:12345
27292024-07-25T05:47:09.063ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 40.633740955s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
27302024-07-25T05:47:11.669ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6389ba5c-9d50-4fe4-b1a2-8be3c6861cb0, kind: Service { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c }, name: Name("ntp-3f311cd5-ee38-4707-ab8a-9331491f8e7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 235])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 4f6a29ee-ca23-4fa8-be69-497060121028, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: b04db43b-dc24-43ce-9d97-32193f88e046, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 2c3f8453-233f-428f-93cc-0bfefb84929d, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: e4efbc31-e513-4290-82f5-c51d8afea0c7, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 221c5e7c-0832-4890-8b0a-f9ba08725876, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4b4998f4-10de-49f2-9326-b482137c5fd3, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: a635cc62-7516-4fd9-b490-62d6de386095, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ac9fa851-e7e1-4295-ac79-d6076526f465, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 1cb96976-2047-44b8-a79c-9fcd74d17bfa, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: a0b9fba0-c330-402f-9bc0-a7510786c98a, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d94d9602-d610-44ce-b9fc-06ded41a1480, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb1f626-0f55-4385-87de-de59052d98d2, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External } } } }, OmicronZoneConfig { id: adff3617-5320-4e42-8b10-1430a1eb0afc, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:434 sled_address = [fd00:1122:3344:101::1]:12345
27312024-07-25T05:47:11.670ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
27322024-07-25T05:47:11.670ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }
27332024-07-25T05:47:11.693ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
27342024-07-25T05:47:11.693ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }
27352024-07-25T05:47:11.713ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
27362024-07-25T05:47:11.713ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }
27372024-07-25T05:47:11.733ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
27382024-07-25T05:47:11.733ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }
27392024-07-25T05:47:11.752ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
27402024-07-25T05:47:11.755ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
27412024-07-25T05:47:11.770ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
27422024-07-25T05:47:11.770ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
27432024-07-25T05:47:11.788ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
27442024-07-25T05:47:11.789ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
27452024-07-25T05:47:11.806ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
27462024-07-25T05:47:11.806ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }
27472024-07-25T05:47:11.825ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
27482024-07-25T05:47:11.825ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }
27492024-07-25T05:47:11.843ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
27502024-07-25T05:47:11.843ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }
27512024-07-25T05:47:11.861ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
27522024-07-25T05:47:11.861ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }
27532024-07-25T05:47:11.880ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
27542024-07-25T05:47:11.880ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }
27552024-07-25T05:47:11.898ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
27562024-07-25T05:47:11.898ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }
27572024-07-25T05:47:11.916ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
27582024-07-25T05:47:11.916ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }
27592024-07-25T05:47:11.934ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
27602024-07-25T05:47:11.935ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }
27612024-07-25T05:47:11.955ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
27622024-07-25T05:47:11.955ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }
27632024-07-25T05:47:11.974ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
27642024-07-25T05:47:11.974ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }
27652024-07-25T05:47:11.993ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
27662024-07-25T05:47:11.993ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }
27672024-07-25T05:47:12.012ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
27682024-07-25T05:47:12.012ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }
27692024-07-25T05:47:12.034ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
27702024-07-25T05:47:12.034ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }
27712024-07-25T05:47:12.053ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
27722024-07-25T05:47:12.053ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:177
27732024-07-25T05:47:12.054ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:177
27742024-07-25T05:47:12.054ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
27752024-07-25T05:47:12.150ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
27762024-07-25T05:47:12.150ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
27772024-07-25T05:47:12.150ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
27782024-07-25T05:47:12.150ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
27792024-07-25T05:47:12.150ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
27802024-07-25T05:47:12.150ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
27812024-07-25T05:47:12.150ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
27822024-07-25T05:47:12.150ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
27832024-07-25T05:47:12.150ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
27842024-07-25T05:47:12.150ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
27852024-07-25T05:47:12.150ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
27862024-07-25T05:47:12.150ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
27872024-07-25T05:47:12.150ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
27882024-07-25T05:47:12.150ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
27892024-07-25T05:47:12.150ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
27902024-07-25T05:47:12.150ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
27912024-07-25T05:47:12.150ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
27922024-07-25T05:47:12.150ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
27932024-07-25T05:47:12.150ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3066 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
27942024-07-25T05:47:12.150ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79", TimeNotSynchronized), ("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d", TimeNotSynchronized), ("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2", TimeNotSynchronized), ("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa", TimeNotSynchronized), ("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7", TimeNotSynchronized), ("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b", TimeNotSynchronized), ("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876", TimeNotSynchronized), ("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862", TimeNotSynchronized), ("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465", TimeNotSynchronized), ("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095", TimeNotSynchronized), ("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c", TimeNotSynchronized), ("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046", TimeNotSynchronized), ("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480", TimeNotSynchronized), ("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e", TimeNotSynchronized), ("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc", TimeNotSynchronized), ("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c", TimeNotSynchronized), ("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9", TimeNotSynchronized), ("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a", TimeNotSynchronized), ("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 479535 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:47495 req_id = 0b7709c4-0e9c-4f77-a1f8-8c61e67f03b3 response_code = 500 uri = /omicron-zones
27952024-07-25T05:47:12.151ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "0b7709c4-0e9c-4f77-a1f8-8c61e67f03b3", "content-length": "124", "date": "Thu, 25 Jul 2024 05:47:11 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "0b7709c4-0e9c-4f77-a1f8-8c61e67f03b3" } file = sled-agent/src/rack_setup/service.rs:472 retry_after = 23.63926763s sled_address = [fd00:1122:3344:101::1]:12345
27962024-07-25T05:47:22.392ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:60831
27972024-07-25T05:47:22.392ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 35 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:60831 req_id = 4933053c-1f7e-4c29-9717-86f054558d6d response_code = 200 uri = /baseboard
27982024-07-25T05:47:35.856ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6389ba5c-9d50-4fe4-b1a2-8be3c6861cb0, kind: Service { id: 3f311cd5-ee38-4707-ab8a-9331491f8e7c }, name: Name("ntp-3f311cd5-ee38-4707-ab8a-9331491f8e7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 235])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 4f6a29ee-ca23-4fa8-be69-497060121028, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: b04db43b-dc24-43ce-9d97-32193f88e046, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 2c3f8453-233f-428f-93cc-0bfefb84929d, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: e4efbc31-e513-4290-82f5-c51d8afea0c7, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 221c5e7c-0832-4890-8b0a-f9ba08725876, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4b4998f4-10de-49f2-9326-b482137c5fd3, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: a635cc62-7516-4fd9-b490-62d6de386095, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ac9fa851-e7e1-4295-ac79-d6076526f465, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 1cb96976-2047-44b8-a79c-9fcd74d17bfa, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: a0b9fba0-c330-402f-9bc0-a7510786c98a, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External } } } }, OmicronZoneConfig { id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d94d9602-d610-44ce-b9fc-06ded41a1480, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External } } } }, OmicronZoneConfig { id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External } } } }, OmicronZoneConfig { id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb1f626-0f55-4385-87de-de59052d98d2, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External } } } }, OmicronZoneConfig { id: adff3617-5320-4e42-8b10-1430a1eb0afc, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:434 sled_address = [fd00:1122:3344:101::1]:12345
27992024-07-25T05:47:35.856ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
28002024-07-25T05:47:35.856ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: aa38c6ce-70ac-4e2b-a383-8201d1e086e8, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: InternalDns }, responder: ... }
28012024-07-25T05:47:35.879ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
28022024-07-25T05:47:35.879ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0ee9b880-7690-4963-bc6b-3dbf4d39892d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: InternalDns }, responder: ... }
28032024-07-25T05:47:35.899ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
28042024-07-25T05:47:35.899ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 23e793cc-4c0e-4aca-92a8-2b30f6dafa5a, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: InternalDns }, responder: ... }
28052024-07-25T05:47:35.916ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
28062024-07-25T05:47:35.916ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4f6a29ee-ca23-4fa8-be69-497060121028, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: CockroachDb }, responder: ... }
28072024-07-25T05:47:35.934ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
28082024-07-25T05:47:35.934ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 3b0743b7-2816-4de6-a40a-d863e46d23f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
28092024-07-25T05:47:35.954ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
28102024-07-25T05:47:35.954ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d835fd87-0881-47dd-8e5b-ebb01fd7b0ce, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
28112024-07-25T05:47:35.972ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
28122024-07-25T05:47:35.972ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 33444b97-3e72-47f7-a319-cbd1ca92b6c8, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
28132024-07-25T05:47:35.991ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
28142024-07-25T05:47:35.991ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4553d8e1-a7ca-4d20-b1a4-394640a4ad4f, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: CockroachDb }, responder: ... }
28152024-07-25T05:47:36.010ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
28162024-07-25T05:47:36.011ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: b04db43b-dc24-43ce-9d97-32193f88e046, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: ExternalDns }, responder: ... }
28172024-07-25T05:47:36.030ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
28182024-07-25T05:47:36.030ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2c3f8453-233f-428f-93cc-0bfefb84929d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: ExternalDns }, responder: ... }
28192024-07-25T05:47:36.051ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
28202024-07-25T05:47:36.051ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 4ba7467e-cf15-4665-9aba-eeedcaaaa862, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Clickhouse }, responder: ... }
28212024-07-25T05:47:36.071ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
28222024-07-25T05:47:36.071ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: a0b9fba0-c330-402f-9bc0-a7510786c98a, dataset_name: DatasetName { pool_name: ZpoolName { id: dd1c81eb-5ddf-42bb-acfe-348d2d141fca (zpool), kind: External }, kind: Crucible }, responder: ... }
28232024-07-25T05:47:36.091ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
28242024-07-25T05:47:36.091ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ee1bae54-08c6-4724-bc97-b5188feb1b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: 59d3cc5a-40b7-44b8-b2af-cfc21755b8bf (zpool), kind: External }, kind: Crucible }, responder: ... }
28252024-07-25T05:47:36.110ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
28262024-07-25T05:47:36.110ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d94d9602-d610-44ce-b9fc-06ded41a1480, dataset_name: DatasetName { pool_name: ZpoolName { id: e78b4f40-df20-4bcf-bd8a-b80afa109d70 (zpool), kind: External }, kind: Crucible }, responder: ... }
28272024-07-25T05:47:36.128ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
28282024-07-25T05:47:36.128ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c, dataset_name: DatasetName { pool_name: ZpoolName { id: 63a12260-6721-4f51-85d5-262aedc90628 (zpool), kind: External }, kind: Crucible }, responder: ... }
28292024-07-25T05:47:36.146ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
28302024-07-25T05:47:36.146ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 7f7f1d81-12f9-4239-8e97-9582c9218b79, dataset_name: DatasetName { pool_name: ZpoolName { id: 587239d7-a145-4bb7-b24a-aff014195deb (zpool), kind: External }, kind: Crucible }, responder: ... }
28312024-07-25T05:47:36.164ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
28322024-07-25T05:47:36.164ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2e3ea66c-7cbd-4cae-aca1-5ebcb142282e, dataset_name: DatasetName { pool_name: ZpoolName { id: 50e63493-6b9c-42de-846b-797a312b5fbc (zpool), kind: External }, kind: Crucible }, responder: ... }
28332024-07-25T05:47:36.182ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
28342024-07-25T05:47:36.182ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: dfb1f626-0f55-4385-87de-de59052d98d2, dataset_name: DatasetName { pool_name: ZpoolName { id: fd75587e-5d72-48df-a44e-8602b2d61469 (zpool), kind: External }, kind: Crucible }, responder: ... }
28352024-07-25T05:47:36.200ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
28362024-07-25T05:47:36.200ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: adff3617-5320-4e42-8b10-1430a1eb0afc, dataset_name: DatasetName { pool_name: ZpoolName { id: 20d995c8-4908-4380-8438-c04a6e3adea2 (zpool), kind: External }, kind: Crucible }, responder: ... }
28372024-07-25T05:47:36.218ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
28382024-07-25T05:47:36.218ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1f722dd3-3360-4e20-99bf-1de5eb86ffd9, dataset_name: DatasetName { pool_name: ZpoolName { id: e43689a1-378e-4be8-9661-3bc522647c6d (zpool), kind: External }, kind: Crucible }, responder: ... }
28392024-07-25T05:47:36.236ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
28402024-07-25T05:47:36.236ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:177
28412024-07-25T05:47:36.237ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:177
28422024-07-25T05:47:36.237ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
28432024-07-25T05:47:36.348ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:258
28442024-07-25T05:47:36.349ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
28452024-07-25T05:47:36.360ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:258
28462024-07-25T05:47:36.360ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
28472024-07-25T05:47:36.395ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:258
28482024-07-25T05:47:36.395ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
28492024-07-25T05:47:36.395ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
28502024-07-25T05:47:36.395ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
28512024-07-25T05:47:36.420ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:258
28522024-07-25T05:47:36.420ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
28532024-07-25T05:47:36.432ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
28542024-07-25T05:47:36.573ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:258
28552024-07-25T05:47:36.574ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
28562024-07-25T05:47:36.574ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
28572024-07-25T05:47:36.600ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
28582024-07-25T05:47:36.600ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
28592024-07-25T05:47:36.600ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
28602024-07-25T05:47:36.600ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
28612024-07-25T05:47:36.600ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
28622024-07-25T05:47:36.600ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c file = illumos-utils/src/zone.rs:316
28632024-07-25T05:47:36.606ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9 file = illumos-utils/src/zone.rs:316
28642024-07-25T05:47:36.612ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480 file = illumos-utils/src/zone.rs:316
28652024-07-25T05:47:36.617ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a file = illumos-utils/src/zone.rs:316
28662024-07-25T05:47:36.623ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79 file = illumos-utils/src/zone.rs:316
28672024-07-25T05:47:36.629ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2 file = illumos-utils/src/zone.rs:316
28682024-07-25T05:47:36.635ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc file = illumos-utils/src/zone.rs:316
28692024-07-25T05:47:36.779ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c file = illumos-utils/src/zone.rs:354
28702024-07-25T05:47:36.784ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9 file = illumos-utils/src/zone.rs:354
28712024-07-25T05:47:36.789ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480 file = illumos-utils/src/zone.rs:354
28722024-07-25T05:47:36.794ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a file = illumos-utils/src/zone.rs:354
28732024-07-25T05:47:36.799ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79 file = illumos-utils/src/zone.rs:354
28742024-07-25T05:47:36.804ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2 file = illumos-utils/src/zone.rs:354
28752024-07-25T05:47:36.809ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc file = illumos-utils/src/zone.rs:354
28762024-07-25T05:47:36.814ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095 file = illumos-utils/src/zone.rs:316
28772024-07-25T05:47:36.819ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c file = illumos-utils/src/zone.rs:316
28782024-07-25T05:47:36.825ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862 file = illumos-utils/src/zone.rs:316
28792024-07-25T05:47:36.831ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3 file = illumos-utils/src/zone.rs:316
28802024-07-25T05:47:36.837ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465 file = illumos-utils/src/zone.rs:316
28812024-07-25T05:47:36.842ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa file = illumos-utils/src/zone.rs:316
28822024-07-25T05:47:36.848ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e file = illumos-utils/src/zone.rs:316
28832024-07-25T05:47:36.891ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:275
28842024-07-25T05:47:36.892ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095 file = illumos-utils/src/zone.rs:354
28852024-07-25T05:47:36.899ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c file = illumos-utils/src/zone.rs:354
28862024-07-25T05:47:36.906ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862 file = illumos-utils/src/zone.rs:354
28872024-07-25T05:47:36.912ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3 file = illumos-utils/src/zone.rs:354
28882024-07-25T05:47:36.920ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465 file = illumos-utils/src/zone.rs:354
28892024-07-25T05:47:36.929ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa file = illumos-utils/src/zone.rs:354
28902024-07-25T05:47:36.936ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e file = illumos-utils/src/zone.rs:354
28912024-07-25T05:47:37.040ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:275
28922024-07-25T05:47:37.163ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:275
28932024-07-25T05:47:37.224ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:275
28942024-07-25T05:47:37.305ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:275
28952024-07-25T05:47:37.307ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:300
28962024-07-25T05:47:37.307ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:192
28972024-07-25T05:47:37.360ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:426 port = Port { inner: PortInner(PortData { name: "opte1", ip: 172.30.2.6, mac: MacAddr6([168, 64, 37, 255, 150, 244]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
28982024-07-25T05:47:37.360ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1207 zone_type = nexus
28992024-07-25T05:47:37.363ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:300
29002024-07-25T05:47:37.363ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:192
29012024-07-25T05:47:37.420ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:426 port = Port { inner: PortInner(PortData { name: "opte2", ip: 172.30.1.5, mac: MacAddr6([168, 64, 37, 255, 194, 154]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
29022024-07-25T05:47:37.420ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1207 zone_type = external_dns
29032024-07-25T05:47:37.423ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:300
29042024-07-25T05:47:37.423ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:192
29052024-07-25T05:47:37.482ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:426 port = Port { inner: PortInner(PortData { name: "opte3", ip: 172.30.1.6, mac: MacAddr6([168, 64, 37, 255, 164, 190]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
29062024-07-25T05:47:37.482ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1207 zone_type = external_dns
29072024-07-25T05:47:37.484ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:300
29082024-07-25T05:47:37.484ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:192
29092024-07-25T05:47:37.544ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:426 port = Port { inner: PortInner(PortData { name: "opte4", ip: 172.30.2.5, mac: MacAddr6([168, 64, 37, 255, 205, 154]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
29102024-07-25T05:47:37.544ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1207 zone_type = nexus
29112024-07-25T05:47:37.546ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:300
29122024-07-25T05:47:37.546ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:192
29132024-07-25T05:47:37.599ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:426 port = Port { inner: PortInner(PortData { name: "opte5", ip: 172.30.2.7, mac: MacAddr6([168, 64, 37, 255, 245, 155]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
29142024-07-25T05:47:37.599ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1207 zone_type = nexus
29152024-07-25T05:47:37.600ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:83
29162024-07-25T05:47:37.600ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:90
29172024-07-25T05:47:37.600ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:83
29182024-07-25T05:47:37.600ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:90
29192024-07-25T05:47:37.603ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:83
29202024-07-25T05:47:37.603ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:90
29212024-07-25T05:47:37.603ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:83
29222024-07-25T05:47:37.603ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:90
29232024-07-25T05:47:37.603ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:83
29242024-07-25T05:47:37.603ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:90
29252024-07-25T05:47:37.605ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:118
29262024-07-25T05:47:37.607ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
29272024-07-25T05:47:37.670ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:118
29282024-07-25T05:47:37.674ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
29292024-07-25T05:47:37.674ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:118
29302024-07-25T05:47:37.679ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:118
29312024-07-25T05:47:37.682ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:118
29322024-07-25T05:47:37.684ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
29332024-07-25T05:47:37.684ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
29342024-07-25T05:47:37.908ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
29352024-07-25T05:47:38.415ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046 file = illumos-utils/src/zone.rs:316
29362024-07-25T05:47:38.471ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876 file = illumos-utils/src/zone.rs:316
29372024-07-25T05:47:38.640ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d file = illumos-utils/src/zone.rs:316
29382024-07-25T05:47:38.676ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b file = illumos-utils/src/zone.rs:316
29392024-07-25T05:47:38.890ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7 file = illumos-utils/src/zone.rs:316
29402024-07-25T05:47:38.970ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046 file = illumos-utils/src/zone.rs:354
29412024-07-25T05:47:39.010ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876 file = illumos-utils/src/zone.rs:354
29422024-07-25T05:47:39.062ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d file = illumos-utils/src/zone.rs:354
29432024-07-25T05:47:39.076ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b file = illumos-utils/src/zone.rs:354
29442024-07-25T05:47:39.086ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7 file = illumos-utils/src/zone.rs:354
29452024-07-25T05:47:41.199ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService10'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::15'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_63a12260-6721-4f51-85d5-262aedc90628/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::15'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='0adc2fba-8696-4e0e-990e-6f991814adbc'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29462024-07-25T05:47:41.200ZINFOSledAgent (ServiceManager): Booting oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c zone file = illumos-utils/src/running_zone.rs:465 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
29472024-07-25T05:47:41.227ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService14'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::16'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_587239d7-a145-4bb7-b24a-aff014195deb/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::16'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='90f8394c-139b-48b3-a256-c34ddcb4cebb'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29482024-07-25T05:47:41.229ZINFOSledAgent (ServiceManager): Booting oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79 zone file = illumos-utils/src/running_zone.rs:465 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
29492024-07-25T05:47:41.246ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService18'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::13'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_59d3cc5a-40b7-44b8-b2af-cfc21755b8bf/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::13'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='a6ba6db4-32f1-4126-9113-00930f444193'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29502024-07-25T05:47:41.247ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService12'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::14'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_e78b4f40-df20-4bcf-bd8a-b80afa109d70/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::14'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='40805802-b334-420b-b3bb-0fa874ba14ba'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29512024-07-25T05:47:41.247ZINFOSledAgent (ServiceManager): Booting oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c zone file = illumos-utils/src/running_zone.rs:465 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
29522024-07-25T05:47:41.259ZINFOSledAgent (ServiceManager): Booting oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480 zone file = illumos-utils/src/running_zone.rs:465 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
29532024-07-25T05:47:41.273ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService23'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::17'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_50e63493-6b9c-42de-846b-797a312b5fbc/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::17'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='d87482a7-c592-4308-a544-0b4e99922e25'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29542024-07-25T05:47:41.273ZINFOSledAgent (ServiceManager): Booting oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e zone file = illumos-utils/src/running_zone.rs:465 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
29552024-07-25T05:47:41.289ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService16'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::19'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_20d995c8-4908-4380-8438-c04a6e3adea2/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::19'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='d9410c88-b362-4eb3-8ebf-90340dafbebd'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29562024-07-25T05:47:41.289ZINFOSledAgent (ServiceManager): Booting oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc zone file = illumos-utils/src/running_zone.rs:465 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
29572024-07-25T05:47:41.315ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService17'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::f'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::f'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29582024-07-25T05:47:41.315ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095 zone file = illumos-utils/src/running_zone.rs:465 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
29592024-07-25T05:47:41.327ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService11'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_e43689a1-378e-4be8-9661-3bc522647c6d/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1a'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='a1bc4f14-42f2-40dd-bd88-fe82fb24cc73'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29602024-07-25T05:47:41.327ZINFOSledAgent (ServiceManager): Booting oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9 zone file = illumos-utils/src/running_zone.rs:465 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
29612024-07-25T05:47:41.422ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService15'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::18'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_fd75587e-5d72-48df-a44e-8602b2d61469/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::18'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='eb003196-f1c8-4f31-8206-37d2bf37f410'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29622024-07-25T05:47:41.433ZINFOSledAgent (ServiceManager): Booting oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2 zone file = illumos-utils/src/running_zone.rs:465 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
29632024-07-25T05:47:41.500ZINFOSledAgent (ServiceManager): Profile for oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService20'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::d'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/oximeter"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="id" value='4b4998f4-10de-49f2-9326-b482137c5fd3'/> <propval type="astring" name="address" value='[fd00:1122:3344:101::d]:12223'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29642024-07-25T05:47:41.500ZINFOSledAgent (ServiceManager): Booting oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3 zone file = illumos-utils/src/running_zone.rs:465 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
29652024-07-25T05:47:41.730ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c") failed: Property not found. retry in 37.426328ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
29662024-07-25T05:47:41.749ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9") failed: Property not found. retry in 54.126915ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
29672024-07-25T05:47:41.767ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480") failed: Property not found. retry in 33.224664ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
29682024-07-25T05:47:41.792ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc") failed: Property not found. retry in 50.529394ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
29692024-07-25T05:47:48.781ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c") failed: Property not found. retry in 77.409831ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
29702024-07-25T05:47:49.099ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79") failed: Property not found. retry in 42.044072ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
29712024-07-25T05:47:49.114ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095") failed: Property not found. retry in 70.216509ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
29722024-07-25T05:47:49.123ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c") failed: Property not found. retry in 52.884266ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
29732024-07-25T05:47:49.144ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e") failed: Property not found. retry in 25.355646ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
29742024-07-25T05:47:49.443ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c") failed: Property not found. retry in 241.507764ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
29752024-07-25T05:47:49.490ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9") failed: Property not found. retry in 145.616899ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
29762024-07-25T05:47:50.536ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480") failed: Property not found. retry in 97.309407ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
29772024-07-25T05:47:50.856ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService13'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::12'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_dd1c81eb-5ddf-42bb-acfe-348d2d141fca/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::12'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='b9f5181b-653b-4495-a501-da2bbf6dd07b'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29782024-07-25T05:47:54.731ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79") failed: Property not found. retry in 51.278324ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
29792024-07-25T05:47:54.777ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2") failed: Property not found. retry in 49.792323ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
29802024-07-25T05:47:54.845ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc") failed: Property not found. retry in 86.940724ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
29812024-07-25T05:47:54.875ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095") failed: Property not found. retry in 95.684907ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
29822024-07-25T05:47:54.894ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:52740
29832024-07-25T05:47:54.905ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 7150 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:52740 req_id = 431a883f-3a90-4063-a4b7-cd143ff04f74 response_code = 200 uri = /baseboard
29842024-07-25T05:47:54.912ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c") failed: Property not found. retry in 125.800962ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
29852024-07-25T05:47:54.949ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3") failed: Property not found. retry in 65.671107ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
29862024-07-25T05:47:54.990ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e") failed: Property not found. retry in 126.539011ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
29872024-07-25T05:47:55.023ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c") failed: Property not found. retry in 265.635286ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
29882024-07-25T05:47:55.024ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService26'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::9'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte3'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::9]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.6:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29892024-07-25T05:47:55.048ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9") failed: Property not found. retry in 140.196785ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
29902024-07-25T05:47:55.066ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480") failed: Property not found. retry in 104.642786ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
29912024-07-25T05:47:55.073ZINFOSledAgent (ServiceManager): Booting oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a zone file = illumos-utils/src/running_zone.rs:465 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
29922024-07-25T05:47:55.096ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79") failed: Property not found. retry in 180.180835ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
29932024-07-25T05:47:55.111ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2") failed: Property not found. retry in 89.701615ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
29942024-07-25T05:47:55.127ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc") failed: Property not found. retry in 208.166526ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
29952024-07-25T05:47:55.143ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095") failed: Property not found. retry in 227.55301ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
29962024-07-25T05:47:55.157ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c") failed: Property not found. retry in 161.885465ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
29972024-07-25T05:47:55.170ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3") failed: Property not found. retry in 84.941899ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
29982024-07-25T05:47:55.182ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e") failed: Property not found. retry in 131.728238ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
29992024-07-25T05:47:55.182ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d zone file = illumos-utils/src/running_zone.rs:465 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
30002024-07-25T05:47:55.210ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9") failed: Property not found. retry in 575.016246ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
30012024-07-25T05:47:55.222ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480") failed: Property not found. retry in 519.11869ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
30022024-07-25T05:47:55.234ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2") failed: Property not found. retry in 238.407015ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
30032024-07-25T05:47:55.235ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService21'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::10'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::10'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
30042024-07-25T05:47:55.237ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService22'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::11'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::11'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
30052024-07-25T05:47:55.241ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465 zone file = illumos-utils/src/running_zone.rs:465 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
30062024-07-25T05:47:55.251ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa zone file = illumos-utils/src/running_zone.rs:465 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
30072024-07-25T05:47:55.272ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3") failed: Property not found. retry in 118.069082ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
30082024-07-25T05:47:55.290ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79") failed: Property not found. retry in 326.503453ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
30092024-07-25T05:47:55.303ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c") failed: Property not found. retry in 503.021503ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
30102024-07-25T05:47:55.326ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e") failed: Property not found. retry in 402.413252ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
30112024-07-25T05:47:55.339ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c") failed: Property not found. retry in 318.710239ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
30122024-07-25T05:47:55.350ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc") failed: Property not found. retry in 417.770481ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
30132024-07-25T05:47:55.386ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095") failed: Property not found. retry in 221.787551ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
30142024-07-25T05:47:55.403ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3") failed: Property not found. retry in 292.956591ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
30152024-07-25T05:47:55.419ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService27'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte1'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
30162024-07-25T05:47:55.420ZINFOSledAgent (ServiceManager): Booting oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b zone file = illumos-utils/src/running_zone.rs:465 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
30172024-07-25T05:47:55.430ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService28'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte4'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
30182024-07-25T05:47:55.441ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d") failed: Property not found. retry in 37.296843ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
30192024-07-25T05:47:55.441ZINFOSledAgent (ServiceManager): Booting oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7 zone file = illumos-utils/src/running_zone.rs:465 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
30202024-07-25T05:47:55.474ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a") failed: Property not found. retry in 53.304346ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
30212024-07-25T05:47:55.485ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2") failed: Property not found. retry in 431.749051ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
30222024-07-25T05:47:55.496ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d") failed: Property not found. retry in 88.66187ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
30232024-07-25T05:47:55.512ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa") failed: Property not found. retry in 54.785529ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
30242024-07-25T05:47:55.513ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService24'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::8'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte2'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::8]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.5:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
30252024-07-25T05:47:57.111ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465") failed: Property not found. retry in 57.268277ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
30262024-07-25T05:47:57.144ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046 zone file = illumos-utils/src/running_zone.rs:465 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
30272024-07-25T05:47:57.186ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc") failed: Property not found. retry in 665.950737ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
30282024-07-25T05:47:57.219ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095") failed: Property not found. retry in 620.479039ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
30292024-07-25T05:47:57.234ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c") failed: Property not found. retry in 1.100537669s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
30302024-07-25T05:47:57.250ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3") failed: Property not found. retry in 903.640511ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
30312024-07-25T05:47:57.274ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465") failed: Property not found. retry in 138.082542ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
30322024-07-25T05:47:57.283ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa") failed: Property not found. retry in 70.413134ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
30332024-07-25T05:47:57.295ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e") failed: Property not found. retry in 795.102851ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
30342024-07-25T05:47:57.305ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c") failed: Property not found. retry in 761.387724ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
30352024-07-25T05:47:57.316ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d") failed: Property not found. retry in 135.910967ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
30362024-07-25T05:47:57.329ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9") failed: Property not found. retry in 463.039605ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
30372024-07-25T05:47:57.341ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b") failed: Property not found. retry in 29.527185ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
30382024-07-25T05:47:57.352ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480") failed: Property not found. retry in 703.862656ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
30392024-07-25T05:47:57.367ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a") failed: Property not found. retry in 55.565974ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
30402024-07-25T05:47:57.378ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79") failed: Property not found. retry in 1.166835065s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
30412024-07-25T05:47:57.388ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7") failed: Property not found. retry in 48.044075ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
30422024-07-25T05:47:57.399ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2") failed: Property not found. retry in 823.606654ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
30432024-07-25T05:47:57.409ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa") failed: Property not found. retry in 114.450589ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
30442024-07-25T05:47:57.420ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b") failed: Property not found. retry in 135.281344ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
30452024-07-25T05:47:57.430ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046") failed: Property not found. retry in 61.128055ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
30462024-07-25T05:47:57.438ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465") failed: Property not found. retry in 289.35517ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
30472024-07-25T05:47:57.449ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a") failed: Property not found. retry in 180.054723ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
30482024-07-25T05:47:57.461ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7") failed: Property not found. retry in 66.844982ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
30492024-07-25T05:47:57.471ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d") failed: Property not found. retry in 565.940208ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
30502024-07-25T05:47:57.503ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046") failed: Property not found. retry in 69.37724ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
30512024-07-25T05:47:57.536ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa") failed: Property not found. retry in 584.99465ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
30522024-07-25T05:47:57.544ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7") failed: Property not found. retry in 291.532203ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
30532024-07-25T05:47:57.567ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b") failed: Property not found. retry in 198.510149ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
30542024-07-25T05:47:57.586ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046") failed: Property not found. retry in 286.656199ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
30552024-07-25T05:47:57.650ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a") failed: Property not found. retry in 324.083036ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
30562024-07-25T05:47:57.740ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465") failed: Property not found. retry in 446.780388ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
30572024-07-25T05:47:57.777ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b") failed: Property not found. retry in 530.244439ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
30582024-07-25T05:47:57.802ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9") failed: Property not found. retry in 1.351110515s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
30592024-07-25T05:47:57.846ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7") failed: Property not found. retry in 268.797536ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
30602024-07-25T05:47:57.856ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095") failed: Property not found. retry in 731.055593ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
30612024-07-25T05:47:57.866ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc") failed: Property not found. retry in 1.377519051s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
30622024-07-25T05:47:57.884ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046") failed: Property not found. retry in 256.241788ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
30632024-07-25T05:47:57.986ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a") failed: Property not found. retry in 1.079940506s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
30642024-07-25T05:47:58.032ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService25'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::c'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte5'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
30652024-07-25T05:47:58.032ZINFOSledAgent (ServiceManager): Booting oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876 zone file = illumos-utils/src/running_zone.rs:465 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
30662024-07-25T05:47:58.054ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d") failed: Property not found. retry in 774.87831ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
30672024-07-25T05:47:58.072ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480") failed: Property not found. retry in 677.574285ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
30682024-07-25T05:47:58.087ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c") failed: Property not found. retry in 1.389192488s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
30692024-07-25T05:47:58.101ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e") failed: Property not found. retry in 1.476488727s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
30702024-07-25T05:47:58.127ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7") failed: Property not found. retry in 1.087282128s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
30712024-07-25T05:47:58.136ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa") failed: Property not found. retry in 1.002843619s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
30722024-07-25T05:47:58.151ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046") failed: Property not found. retry in 1.188552694s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
30732024-07-25T05:47:58.166ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3") failed: Property not found. retry in 760.161627ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
30742024-07-25T05:47:58.198ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465") failed: Property not found. retry in 752.772191ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
30752024-07-25T05:47:58.236ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2") failed: Property not found. retry in 593.833015ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
30762024-07-25T05:47:58.260ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876") failed: Property not found. retry in 63.719909ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
30772024-07-25T05:47:58.320ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b") failed: Property not found. retry in 626.855082ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
30782024-07-25T05:47:58.333ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876") failed: Property not found. retry in 145.846146ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
30792024-07-25T05:47:58.362ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c") failed: Property not found. retry in 905.236348ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
30802024-07-25T05:47:58.494ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876") failed: Property not found. retry in 163.363908ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
30812024-07-25T05:47:58.557ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79") failed: Property not found. retry in 1.177290733s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
30822024-07-25T05:47:58.602ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095") failed: Property not found. retry in 603.629053ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
30832024-07-25T05:47:58.674ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876") failed: Property not found. retry in 569.340709ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
30842024-07-25T05:47:58.764ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480") failed: Property not found. retry in 880.187151ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
30852024-07-25T05:47:58.843ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d") failed: Property not found. retry in 1.09792147s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
30862024-07-25T05:47:58.854ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2") failed: Property not found. retry in 833.595025ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
30872024-07-25T05:47:58.939ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3") failed: Property not found. retry in 1.227164237s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
30882024-07-25T05:47:58.962ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b") failed: Property not found. retry in 615.973401ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
30892024-07-25T05:47:58.974ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465") failed: Property not found. retry in 1.192478695s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
30902024-07-25T05:47:59.083ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a") failed: Property not found. retry in 866.172582ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
30912024-07-25T05:47:59.154ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa") failed: Property not found. retry in 610.016509ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
30922024-07-25T05:47:59.168ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9") failed: Property not found. retry in 1.381868421s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
30932024-07-25T05:47:59.221ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095") failed: Property not found. retry in 946.164709ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
30942024-07-25T05:47:59.236ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7") failed: Property not found. retry in 879.632144ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
30952024-07-25T05:47:59.259ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc") failed: Property not found. retry in 1.206140776s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
30962024-07-25T05:47:59.276ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876") failed: Property not found. retry in 959.495122ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
30972024-07-25T05:47:59.296ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c") failed: Property not found. retry in 794.853114ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
30982024-07-25T05:47:59.354ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046") failed: Property not found. retry in 670.679984ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
30992024-07-25T05:47:59.493ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c") failed: Property not found. retry in 698.874985ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
31002024-07-25T05:47:59.598ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b") failed: Property not found. retry in 647.971118ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
31012024-07-25T05:47:59.618ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e") failed: Property not found. retry in 1.095156255s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
31022024-07-25T05:47:59.660ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480") failed: Property not found. retry in 569.621801ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
31032024-07-25T05:47:59.709ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2") failed: Property not found. retry in 1.424332475s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
31042024-07-25T05:47:59.754ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79") failed: Property not found. retry in 1.457034465s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
31052024-07-25T05:47:59.782ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa") failed: Property not found. retry in 1.398853443s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
31062024-07-25T05:47:59.958ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d") failed: Property not found. retry in 1.411180647s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
31072024-07-25T05:47:59.973ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a") failed: Property not found. retry in 674.383159ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
31082024-07-25T05:48:00.042ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046") failed: Property not found. retry in 1.332190553s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
31092024-07-25T05:48:00.106ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c") failed: Property not found. retry in 742.420306ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
31102024-07-25T05:48:00.131ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7") failed: Property not found. retry in 839.973084ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
31112024-07-25T05:48:00.179ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3") failed: Property not found. retry in 1.120226528s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
31122024-07-25T05:48:00.193ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465") failed: Property not found. retry in 1.140134864s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
31132024-07-25T05:48:00.206ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c") failed: Property not found. retry in 582.304513ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
31142024-07-25T05:48:00.217ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095") failed: Property not found. retry in 825.450297ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
31152024-07-25T05:48:00.242ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480") failed: Property not found. retry in 1.008307029s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
31162024-07-25T05:48:00.254ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876") failed: Property not found. retry in 977.954121ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
31172024-07-25T05:48:00.268ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b") failed: Property not found. retry in 731.029008ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
31182024-07-25T05:48:00.480ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc") failed: Property not found. retry in 1.299031098s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
31192024-07-25T05:48:00.566ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9") failed: Property not found. retry in 918.154749ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
31202024-07-25T05:48:00.667ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a") failed: Property not found. retry in 1.089652177s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
31212024-07-25T05:48:00.732ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e") failed: Property not found. retry in 681.033618ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
31222024-07-25T05:48:00.802ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c") failed: Property not found. retry in 1.418942366s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
31232024-07-25T05:48:00.868ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c") failed: Property not found. retry in 808.892026ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
31242024-07-25T05:48:00.987ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7") failed: Property not found. retry in 649.61465ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
31252024-07-25T05:48:01.016ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b") failed: Property not found. retry in 1.068968292s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
31262024-07-25T05:48:01.055ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095") failed: Property not found. retry in 1.420865688s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
31272024-07-25T05:48:01.147ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2") failed: Property not found. retry in 1.480647827s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
31282024-07-25T05:48:01.193ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa") failed: Property not found. retry in 561.875433ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
31292024-07-25T05:48:01.224ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79") failed: Property not found. retry in 627.010881ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
31302024-07-25T05:48:01.245ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876") failed: Property not found. retry in 1.362808117s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
31312024-07-25T05:48:01.264ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480") failed: Property not found. retry in 810.424128ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
31322024-07-25T05:48:01.310ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3") failed: Property not found. retry in 1.149008714s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
31332024-07-25T05:48:01.344ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465") failed: Property not found. retry in 520.589744ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
31342024-07-25T05:48:01.383ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d") failed: Property not found. retry in 1.323923045s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
31352024-07-25T05:48:01.395ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046") failed: Property not found. retry in 1.450370551s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
31362024-07-25T05:48:01.425ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e") failed: Property not found. retry in 1.476939112s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
31372024-07-25T05:48:01.498ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9") failed: Property not found. retry in 773.808732ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
31382024-07-25T05:48:01.649ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7") failed: Property not found. retry in 1.067806629s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
31392024-07-25T05:48:01.692ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c") failed: Property not found. retry in 1.497117333s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
31402024-07-25T05:48:01.721ZINFOSledAgent (ServiceManager): Profile for oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService19'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::e'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::e'/> <propval type="astring" name="listen_port" value='8123'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
31412024-07-25T05:48:01.729ZINFOSledAgent (ServiceManager): Booting oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862 zone file = illumos-utils/src/running_zone.rs:465 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
31422024-07-25T05:48:01.769ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa") failed: Property not found. retry in 628.312703ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
31432024-07-25T05:48:01.780ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a") failed: Property not found. retry in 594.785266ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
31442024-07-25T05:48:01.789ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc") failed: Property not found. retry in 545.382437ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
31452024-07-25T05:48:01.863ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79") failed: Property not found. retry in 1.142089643s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
31462024-07-25T05:48:01.877ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465") failed: Property not found. retry in 530.970014ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
31472024-07-25T05:48:01.975ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862") failed: Property not found. retry in 68.156145ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
31482024-07-25T05:48:02.056ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862") failed: Property not found. retry in 118.316144ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
31492024-07-25T05:48:02.087ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480") failed: Property not found. retry in 702.761469ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
31502024-07-25T05:48:02.099ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b") failed: Property not found. retry in 1.311457962s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
31512024-07-25T05:48:02.211ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862") failed: Property not found. retry in 195.672498ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
31522024-07-25T05:48:02.234ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c") failed: Property not found. retry in 1.330497282s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
31532024-07-25T05:48:02.289ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9") failed: Property not found. retry in 1.239252337s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
31542024-07-25T05:48:02.345ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc") failed: Property not found. retry in 1.224978233s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
31552024-07-25T05:48:02.384ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a") failed: Property not found. retry in 830.877686ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
31562024-07-25T05:48:02.406ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa") failed: Property not found. retry in 806.312745ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
31572024-07-25T05:48:02.416ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862") failed: Property not found. retry in 509.376192ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
31582024-07-25T05:48:02.424ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465") failed: Property not found. retry in 1.0337136s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
31592024-07-25T05:48:02.467ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3") failed: Property not found. retry in 1.131608349s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
31602024-07-25T05:48:02.484ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095") failed: Property not found. retry in 1.260962384s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
31612024-07-25T05:48:02.618ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876") failed: Property not found. retry in 1.450966256s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
31622024-07-25T05:48:02.637ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2") failed: Property not found. retry in 566.117525ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
31632024-07-25T05:48:02.716ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d") failed: Property not found. retry in 1.151011803s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
31642024-07-25T05:48:02.725ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7") failed: Property not found. retry in 833.8049ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
31652024-07-25T05:48:02.799ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480") failed: Property not found. retry in 1.145716971s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
31662024-07-25T05:48:02.855ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046") failed: Property not found. retry in 970.076581ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
31672024-07-25T05:48:02.912ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e") failed: Property not found. retry in 1.19940143s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
31682024-07-25T05:48:02.935ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862") failed: Property not found. retry in 803.626666ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
31692024-07-25T05:48:03.016ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79") failed: Property not found. retry in 1.139490235s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
31702024-07-25T05:48:03.201ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c") failed: Property not found. retry in 982.621469ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
31712024-07-25T05:48:03.213ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2") failed: Property not found. retry in 956.239748ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
31722024-07-25T05:48:03.222ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa") failed: Property not found. retry in 1.453292592s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
31732024-07-25T05:48:03.230ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a") failed: Property not found. retry in 1.347344347s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
31742024-07-25T05:48:03.420ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b") failed: Property not found. retry in 659.865857ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
31752024-07-25T05:48:03.468ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465") failed: Property not found. retry in 529.799556ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
31762024-07-25T05:48:03.537ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9") failed: Property not found. retry in 838.589487ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
31772024-07-25T05:48:03.568ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7") failed: Property not found. retry in 893.170702ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
31782024-07-25T05:48:03.576ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c") failed: Property not found. retry in 701.028738ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
31792024-07-25T05:48:03.584ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc") failed: Property not found. retry in 1.390167238s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
31802024-07-25T05:48:03.607ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3") failed: Property not found. retry in 963.947665ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
31812024-07-25T05:48:03.749ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862") failed: Property not found. retry in 627.344597ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
31822024-07-25T05:48:03.757ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095") failed: Property not found. retry in 1.027407224s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
31832024-07-25T05:48:03.834ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046") failed: Property not found. retry in 1.367442835s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
31842024-07-25T05:48:03.878ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d") failed: Property not found. retry in 625.428331ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
31852024-07-25T05:48:03.956ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480") failed: Property not found. retry in 805.40977ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
31862024-07-25T05:48:04.007ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465") failed: Property not found. retry in 587.73506ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
31872024-07-25T05:48:04.080ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876") failed: Property not found. retry in 825.721099ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
31882024-07-25T05:48:04.091ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b") failed: Property not found. retry in 1.152680622s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
31892024-07-25T05:48:04.121ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e") failed: Property not found. retry in 1.005571705s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
31902024-07-25T05:48:04.164ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79") failed: Property not found. retry in 1.481577477s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
31912024-07-25T05:48:04.179ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2") failed: Property not found. retry in 640.06715ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
31922024-07-25T05:48:04.192ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c") failed: Property not found. retry in 1.408361753s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
31932024-07-25T05:48:04.289ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c") failed: Property not found. retry in 583.747966ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
31942024-07-25T05:48:04.387ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9") failed: Property not found. retry in 588.546793ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
31952024-07-25T05:48:04.395ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862") failed: Property not found. retry in 1.476708464s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
31962024-07-25T05:48:04.471ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7") failed: Property not found. retry in 786.984016ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
31972024-07-25T05:48:04.516ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d") failed: Property not found. retry in 582.628446ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
31982024-07-25T05:48:04.584ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3") failed: Property not found. retry in 1.443464523s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
31992024-07-25T05:48:04.598ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a") failed: Property not found. retry in 1.185153166s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
32002024-07-25T05:48:04.608ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465") failed: Property not found. retry in 1.244441889s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
32012024-07-25T05:48:04.690ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa") failed: Property not found. retry in 920.542968ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
32022024-07-25T05:48:04.774ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480") failed: Property not found. retry in 547.071478ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
32032024-07-25T05:48:04.796ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095") failed: Property not found. retry in 1.43137454s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
32042024-07-25T05:48:04.831ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2") failed: Property not found. retry in 598.410726ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
32052024-07-25T05:48:04.886ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c") failed: Property not found. retry in 654.9178ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
32062024-07-25T05:48:04.921ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876") failed: Property not found. retry in 1.384886378s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
32072024-07-25T05:48:04.997ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc") failed: Property not found. retry in 1.391208911s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
32082024-07-25T05:48:05.013ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9") failed: Property not found. retry in 613.393346ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
32092024-07-25T05:48:05.115ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d") failed: Property not found. retry in 1.346065907s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
32102024-07-25T05:48:05.144ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e") failed: Property not found. retry in 953.877434ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
32112024-07-25T05:48:05.221ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046") failed: Property not found. retry in 512.365712ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
32122024-07-25T05:48:05.262ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b") failed: Property not found. retry in 852.959817ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
32132024-07-25T05:48:05.278ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7") failed: Property not found. retry in 594.378212ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
32142024-07-25T05:48:05.342ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480") failed: Property not found. retry in 1.238066207s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
32152024-07-25T05:48:05.445ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2") failed: Property not found. retry in 1.315565091s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
32162024-07-25T05:48:05.559ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c") failed: Property not found. retry in 629.877845ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
32172024-07-25T05:48:05.616ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c") failed: Property not found. retry in 970.972461ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
32182024-07-25T05:48:05.630ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa") failed: Property not found. retry in 1.155659569s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
32192024-07-25T05:48:05.644ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9") failed: Property not found. retry in 794.259435ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
32202024-07-25T05:48:05.661ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79") failed: Property not found. retry in 540.503761ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
32212024-07-25T05:48:05.750ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046") failed: Property not found. retry in 1.295600662s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
32222024-07-25T05:48:05.798ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a") failed: Property not found. retry in 1.006393582s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
32232024-07-25T05:48:05.869ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465") failed: Property not found. retry in 756.727781ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
32242024-07-25T05:48:05.886ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7") failed: Property not found. retry in 1.011246548s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
32252024-07-25T05:48:05.900ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862") failed: Property not found. retry in 1.030796806s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
32262024-07-25T05:48:06.045ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3") failed: Property not found. retry in 661.425914ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
32272024-07-25T05:48:06.115ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e") failed: Property not found. retry in 1.003564486s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
32282024-07-25T05:48:06.131ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b") failed: Property not found. retry in 528.806356ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
32292024-07-25T05:48:06.203ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c") failed: Property not found. retry in 778.027565ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
32302024-07-25T05:48:06.217ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79") failed: Property not found. retry in 888.080612ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
32312024-07-25T05:48:06.244ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095") failed: Property not found. retry in 1.258616118s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
32322024-07-25T05:48:06.318ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876") failed: Property not found. retry in 1.205911159s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
32332024-07-25T05:48:06.400ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc") failed: Property not found. retry in 1.257067845s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
32342024-07-25T05:48:06.451ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9") failed: Property not found. retry in 1.29192886s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
32352024-07-25T05:48:06.474ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d") failed: Property not found. retry in 1.328237469s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
32362024-07-25T05:48:06.607ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c") failed: Property not found. retry in 839.872034ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
32372024-07-25T05:48:06.637ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465") failed: Property not found. retry in 928.024179ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
32382024-07-25T05:48:06.672ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b") failed: Property not found. retry in 1.143472534s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
32392024-07-25T05:48:06.719ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3") failed: Property not found. retry in 1.43169962s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
32402024-07-25T05:48:06.771ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2") failed: Property not found. retry in 566.220388ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
32412024-07-25T05:48:06.797ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa") failed: Property not found. retry in 1.138022568s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
32422024-07-25T05:48:06.815ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a") failed: Property not found. retry in 1.27333666s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
32432024-07-25T05:48:06.907ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7") failed: Property not found. retry in 1.335819115s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
32442024-07-25T05:48:06.941ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862") failed: Property not found. retry in 1.458677163s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
32452024-07-25T05:48:06.990ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c") failed: Property not found. retry in 714.797094ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
32462024-07-25T05:48:07.055ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046") failed: Property not found. retry in 1.475182936s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
32472024-07-25T05:48:07.114ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79") failed: Property not found. retry in 1.122916619s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
32482024-07-25T05:48:07.129ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e") failed: Property not found. retry in 1.408205128s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
32492024-07-25T05:48:07.605ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095") failed: Property not found. retry in 1.398048992s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
32502024-07-25T05:48:07.613ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c") failed: Property not found. retry in 865.910902ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
32512024-07-25T05:48:07.622ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876") failed: Property not found. retry in 1.434145982s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
32522024-07-25T05:48:07.630ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465") failed: Property not found. retry in 544.577551ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
32532024-07-25T05:48:07.666ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc") failed: Property not found. retry in 1.297798619s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
32542024-07-25T05:48:07.713ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c") failed: Property not found. retry in 1.003862399s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
32552024-07-25T05:48:07.751ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9") failed: Property not found. retry in 750.815675ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
32562024-07-25T05:48:07.810ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d") failed: Property not found. retry in 1.186019981s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
32572024-07-25T05:48:07.825ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b") failed: Property not found. retry in 797.104249ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
32582024-07-25T05:48:07.944ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa") failed: Property not found. retry in 673.62495ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
32592024-07-25T05:48:08.098ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a") failed: Property not found. retry in 1.164813402s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
32602024-07-25T05:48:08.160ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3") failed: Property not found. retry in 540.098897ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
32612024-07-25T05:48:08.184ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465") failed: Property not found. retry in 1.036108339s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
32622024-07-25T05:48:08.246ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79") failed: Property not found. retry in 953.191883ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
32632024-07-25T05:48:08.255ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7") failed: Property not found. retry in 1.346235289s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
32642024-07-25T05:48:08.411ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862") failed: Property not found. retry in 820.340184ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
32652024-07-25T05:48:08.490ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c") failed: Property not found. retry in 1.158796691s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
32662024-07-25T05:48:08.511ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9") failed: Property not found. retry in 1.4604962s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
32672024-07-25T05:48:08.540ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046") failed: Property not found. retry in 1.095409857s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
32682024-07-25T05:48:08.549ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e") failed: Property not found. retry in 725.853315ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
32692024-07-25T05:48:08.627ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa") failed: Property not found. retry in 1.468642945s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
32702024-07-25T05:48:08.635ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b") failed: Property not found. retry in 530.000998ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
32712024-07-25T05:48:08.711ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3") failed: Property not found. retry in 1.339465622s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
32722024-07-25T05:48:08.726ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c") failed: Property not found. retry in 1.42021812s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
32732024-07-25T05:48:08.976ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc") failed: Property not found. retry in 708.308815ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
32742024-07-25T05:48:09.008ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d") failed: Property not found. retry in 1.322168562s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
32752024-07-25T05:48:09.015ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095") failed: Property not found. retry in 833.191167ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
32762024-07-25T05:48:09.066ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876") failed: Property not found. retry in 1.475454757s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
32772024-07-25T05:48:09.176ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b") failed: Property not found. retry in 518.39515ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
32782024-07-25T05:48:09.209ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79") failed: Property not found. retry in 1.373480999s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
32792024-07-25T05:48:09.229ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465") failed: Property not found. retry in 1.468661622s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
32802024-07-25T05:48:09.240ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862") failed: Property not found. retry in 1.435637814s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
32812024-07-25T05:48:09.273ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a") failed: Property not found. retry in 1.395025414s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
32822024-07-25T05:48:09.284ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e") failed: Property not found. retry in 748.933897ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
32832024-07-25T05:48:09.614ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7") failed: Property not found. retry in 1.23586947s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
32842024-07-25T05:48:09.646ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046") failed: Property not found. retry in 1.311832728s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
32852024-07-25T05:48:09.658ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c") failed: Property not found. retry in 1.287324151s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
32862024-07-25T05:48:09.695ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc") failed: Property not found. retry in 819.573776ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
32872024-07-25T05:48:09.703ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b") failed: Property not found. retry in 1.15383376s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
32882024-07-25T05:48:09.859ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095") failed: Property not found. retry in 1.424088924s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
32892024-07-25T05:48:09.982ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9") failed: Property not found. retry in 661.418982ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
32902024-07-25T05:48:10.044ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e") failed: Property not found. retry in 513.444686ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
32912024-07-25T05:48:10.061ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3") failed: Property not found. retry in 574.842712ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
32922024-07-25T05:48:10.106ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa") failed: Property not found. retry in 1.233645214s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
32932024-07-25T05:48:10.156ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c") failed: Property not found. retry in 1.460191433s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
32942024-07-25T05:48:10.185ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 92.760311014s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
32952024-07-25T05:48:10.340ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d") failed: Property not found. retry in 802.070966ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
32962024-07-25T05:48:10.526ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc") failed: Property not found. retry in 741.628336ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
32972024-07-25T05:48:10.552ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876") failed: Property not found. retry in 851.590942ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
32982024-07-25T05:48:10.592ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79") failed: Property not found. retry in 691.892413ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
32992024-07-25T05:48:10.676ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a") failed: Property not found. retry in 944.940253ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
33002024-07-25T05:48:10.684ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862") failed: Property not found. retry in 909.886088ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
33012024-07-25T05:48:10.706ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465") failed: Property not found. retry in 1.239706361s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
33022024-07-25T05:48:11.612ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862") failed: Property not found. retry in 656.074483ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
33032024-07-25T05:48:12.288ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862") failed: Property not found. retry in 591.370629ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
33042024-07-25T05:48:12.903ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862") failed: Property not found. retry in 1.068722819s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
33052024-07-25T05:48:13.993ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862") failed: Property not found. retry in 934.474786ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
33062024-07-25T05:48:14.638ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/8fa4bc78-1026-41f0-b09d-c67224071376): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:225 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
33072024-07-25T05:48:14.638ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/8fa4bc78-1026-41f0-b09d-c67224071376): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:619 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
33082024-07-25T05:48:14.942ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862") failed: Property not found. retry in 799.660427ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
33092024-07-25T05:48:15.159ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/8fa4bc78-1026-41f0-b09d-c67224071376): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:619 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
33102024-07-25T05:48:15.642ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/8fa4bc78-1026-41f0-b09d-c67224071376): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:225 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
33112024-07-25T05:48:15.772ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3062 zone = oxz_crucible_655516f5-b8a5-4e1d-b3e9-1b4fd5e5717c
33122024-07-25T05:48:15.772ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3062 zone = oxz_external_dns_2c3f8453-233f-428f-93cc-0bfefb84929d
33132024-07-25T05:48:15.772ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3062 zone = oxz_crucible_1f722dd3-3360-4e20-99bf-1de5eb86ffd9
33142024-07-25T05:48:15.772ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3062 zone = oxz_nexus_85472347-dfd9-425e-bd98-9b81ebdf3b2b
33152024-07-25T05:48:15.772ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3062 zone = oxz_crucible_d94d9602-d610-44ce-b9fc-06ded41a1480
33162024-07-25T05:48:15.772ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3062 zone = oxz_crucible_a0b9fba0-c330-402f-9bc0-a7510786c98a
33172024-07-25T05:48:15.772ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3062 zone = oxz_crucible_7f7f1d81-12f9-4239-8e97-9582c9218b79
33182024-07-25T05:48:15.772ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3062 zone = oxz_nexus_e4efbc31-e513-4290-82f5-c51d8afea0c7
33192024-07-25T05:48:15.772ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3062 zone = oxz_crucible_dfb1f626-0f55-4385-87de-de59052d98d2
33202024-07-25T05:48:15.772ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3062 zone = oxz_external_dns_b04db43b-dc24-43ce-9d97-32193f88e046
33212024-07-25T05:48:15.772ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3062 zone = oxz_crucible_adff3617-5320-4e42-8b10-1430a1eb0afc
33222024-07-25T05:48:15.772ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3062 zone = oxz_crucible_pantry_a635cc62-7516-4fd9-b490-62d6de386095
33232024-07-25T05:48:15.772ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3062 zone = oxz_crucible_ee1bae54-08c6-4724-bc97-b5188feb1b8c
33242024-07-25T05:48:15.772ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3062 zone = oxz_nexus_221c5e7c-0832-4890-8b0a-f9ba08725876
33252024-07-25T05:48:15.772ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3062 zone = oxz_clickhouse_4ba7467e-cf15-4665-9aba-eeedcaaaa862
33262024-07-25T05:48:15.772ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3062 zone = oxz_oximeter_4b4998f4-10de-49f2-9326-b482137c5fd3
33272024-07-25T05:48:15.772ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3062 zone = oxz_crucible_pantry_ac9fa851-e7e1-4295-ac79-d6076526f465
33282024-07-25T05:48:15.773ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3062 zone = oxz_crucible_pantry_1cb96976-2047-44b8-a79c-9fcd74d17bfa
33292024-07-25T05:48:15.773ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3062 zone = oxz_crucible_2e3ea66c-7cbd-4cae-aca1-5ebcb142282e
33302024-07-25T05:48:15.774ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
33312024-07-25T05:48:15.777ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
33322024-07-25T05:48:15.781ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 39919954 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:47495 req_id = 919d718d-e5d0-47cf-8e40-ace8413870d3 response_code = 204 uri = /omicron-zones
33332024-07-25T05:48:15.793ZINFOSledAgent (RSS): Finished setting up services file = sled-agent/src/rack_setup/service.rs:1247
33342024-07-25T05:48:15.793ZINFOSledAgent (RSS): Writing ledger to /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
33352024-07-25T05:48:15.793ZINFOSledAgent (RSS): Writing ledger to /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
33362024-07-25T05:48:15.796ZINFOSledAgent (RSS): Handing off control to Nexus file = sled-agent/src/rack_setup/service.rs:684
33372024-07-25T05:48:15.798ZINFOSledAgent (RSS): Nexus address: [fd00:1122:3344:101::c]:12221 file = sled-agent/src/rack_setup/service.rs:697
33382024-07-25T05:48:15.829ZINFOSledAgent (RSS): rack_network_config: RackNetworkConfigV2 { bfd: [], bgp: [], infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV2 { addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.1.174, width: 24, }, ), vlan_id: None, }, ], autoneg: false, bgp_peers: [], port: "qsfp0", routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.1.100, vlan_id: None, }, ], switch: Switch0, uplink_port_fec: None, uplink_port_speed: Speed40G, }, ], rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, } file = sled-agent/src/rack_setup/service.rs:828
33392024-07-25T05:48:15.831ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b4c6410d-62e1-4369-a8c0-f8e6399dd808/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:890
33402024-07-25T05:48:15.941ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b4c6410d-62e1-4369-a8c0-f8e6399dd808/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:890
33412024-07-25T05:48:16.047ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b4c6410d-62e1-4369-a8c0-f8e6399dd808/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:890
33422024-07-25T05:48:16.130ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b4c6410d-62e1-4369-a8c0-f8e6399dd808/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:890
33432024-07-25T05:48:16.318ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b4c6410d-62e1-4369-a8c0-f8e6399dd808/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:890
33442024-07-25T05:48:16.610ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b4c6410d-62e1-4369-a8c0-f8e6399dd808/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:890
33452024-07-25T05:48:16.644ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/8fa4bc78-1026-41f0-b09d-c67224071376): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:225 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
33462024-07-25T05:48:16.904ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b4c6410d-62e1-4369-a8c0-f8e6399dd808/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:890
33472024-07-25T05:48:17.156ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/8fa4bc78-1026-41f0-b09d-c67224071376): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:619 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
33482024-07-25T05:48:17.221ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b4c6410d-62e1-4369-a8c0-f8e6399dd808/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:890
33492024-07-25T05:48:17.580ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b4c6410d-62e1-4369-a8c0-f8e6399dd808/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:890
33502024-07-25T05:48:17.647ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/8fa4bc78-1026-41f0-b09d-c67224071376): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:225 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
33512024-07-25T05:48:17.888ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b4c6410d-62e1-4369-a8c0-f8e6399dd808/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:890
33522024-07-25T05:48:18.199ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b4c6410d-62e1-4369-a8c0-f8e6399dd808/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:890
33532024-07-25T05:48:18.551ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b4c6410d-62e1-4369-a8c0-f8e6399dd808/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:890
33542024-07-25T05:48:18.650ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/8fa4bc78-1026-41f0-b09d-c67224071376): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:225 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
33552024-07-25T05:48:19.062ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b4c6410d-62e1-4369-a8c0-f8e6399dd808/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:890
33562024-07-25T05:48:19.157ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/8fa4bc78-1026-41f0-b09d-c67224071376): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:619 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
33572024-07-25T05:48:19.654ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/8fa4bc78-1026-41f0-b09d-c67224071376): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:225 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
33582024-07-25T05:48:20.215ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b4c6410d-62e1-4369-a8c0-f8e6399dd808/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:890
33592024-07-25T05:48:20.657ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/8fa4bc78-1026-41f0-b09d-c67224071376): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:225 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
33602024-07-25T05:48:21.156ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/8fa4bc78-1026-41f0-b09d-c67224071376): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:619 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
33612024-07-25T05:48:21.337ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b4c6410d-62e1-4369-a8c0-f8e6399dd808/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:890
33622024-07-25T05:48:21.659ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/8fa4bc78-1026-41f0-b09d-c67224071376): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:225 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
33632024-07-25T05:48:22.583ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b4c6410d-62e1-4369-a8c0-f8e6399dd808/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:890
33642024-07-25T05:48:22.663ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/8fa4bc78-1026-41f0-b09d-c67224071376): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:225 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
33652024-07-25T05:48:23.156ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/8fa4bc78-1026-41f0-b09d-c67224071376): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:619 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
33662024-07-25T05:48:23.665ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/8fa4bc78-1026-41f0-b09d-c67224071376): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:225 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
33672024-07-25T05:48:24.135ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "598fdfcb-1e35-4345-9c99-c06ae105979b", "content-length": "173", "date": "Thu, 25 Jul 2024 05:48:24 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: rack with id \"b4c6410d-62e1-4369-a8c0-f8e6399dd808\"", request_id: "598fdfcb-1e35-4345-9c99-c06ae105979b" } file = sled-agent/src/rack_setup/service.rs:890
33682024-07-25T05:48:24.948ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:39843
33692024-07-25T05:48:24.948ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 59 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:39843 req_id = ee157287-dfd9-4f15-a1c9-c673da533b35 response_code = 200 uri = /baseboard
33702024-07-25T05:48:26.532ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "950bc629-e843-4819-95da-4705da84e502", "content-length": "173", "date": "Thu, 25 Jul 2024 05:48:26 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"8fa4bc78-1026-41f0-b09d-c67224071376\"", request_id: "950bc629-e843-4819-95da-4705da84e502" } file = sled-agent/src/rack_setup/service.rs:890
33712024-07-25T05:48:27.194ZINFOSledAgent (NexusNotifierTask): Successfully put SledAgentInfo to nexus file = sled-agent/src/nexus.rs:592 new_generation = 1 old_generation = 1 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
33722024-07-25T05:48:27.326ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:61068
33732024-07-25T05:48:27.327ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
33742024-07-25T05:48:27.415ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 88886 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:61068 req_id = d672f128-99e9-4e36-beae-65884ed4d4f9 response_code = 200 uri = /inventory
33752024-07-25T05:48:27.416ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
33762024-07-25T05:48:27.416ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:177
33772024-07-25T05:48:27.417ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:177
33782024-07-25T05:48:27.418ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 2562 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:61068 req_id = 463d6462-bdb6-4647-b7a6-310b06d1b2d8 response_code = 200 uri = /omicron-zones
33792024-07-25T05:48:28.635ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:44913
33802024-07-25T05:48:28.638ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:673 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
33812024-07-25T05:48:28.638ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte0" rules = []
33822024-07-25T05:48:28.638ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
33832024-07-25T05:48:28.638ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
33842024-07-25T05:48:28.638ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
33852024-07-25T05:48:28.638ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
33862024-07-25T05:48:28.638ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
33872024-07-25T05:48:28.638ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 2954 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:44913 req_id = 2c2fb3cd-7dcd-4d96-921b-6b0af06f799d response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
33882024-07-25T05:48:28.641ZINFOSledAgent (RSS): Handoff to Nexus is complete file = sled-agent/src/rack_setup/service.rs:900
33892024-07-25T05:48:28.892ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:45601
33902024-07-25T05:48:28.892ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
33912024-07-25T05:48:28.969ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 77020 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:45601 req_id = 3edc927e-750e-419d-b536-465d019f7bff response_code = 200 uri = /inventory
33922024-07-25T05:48:28.969ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
33932024-07-25T05:48:28.969ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/config/omicron-zones.json file = common/src/ledger.rs:177
33942024-07-25T05:48:28.970ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/config/omicron-zones.json file = common/src/ledger.rs:177
33952024-07-25T05:48:28.971ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 1594 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:45601 req_id = 3032e27f-7003-47f1-85b5-d5fd8a005c60 response_code = 200 uri = /omicron-zones
33962024-07-25T05:48:29.900ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:53469
33972024-07-25T05:48:29.901ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:673 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
33982024-07-25T05:48:29.901ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte0" rules = []
33992024-07-25T05:48:29.901ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34002024-07-25T05:48:29.901ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34012024-07-25T05:48:29.901ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
34022024-07-25T05:48:29.901ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
34032024-07-25T05:48:29.901ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34042024-07-25T05:48:29.901ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 379 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:53469 req_id = 36f6d103-f86a-4733-ab6f-55b295a88e52 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
34052024-07-25T05:48:30.023ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:47040
34062024-07-25T05:48:30.024ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:673 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
34072024-07-25T05:48:30.024ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte0" rules = []
34082024-07-25T05:48:30.024ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34092024-07-25T05:48:30.024ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34102024-07-25T05:48:30.024ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
34112024-07-25T05:48:30.024ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
34122024-07-25T05:48:30.024ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34132024-07-25T05:48:30.024ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 603 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:47040 req_id = 8bdcc0f4-a25b-4d91-8d7f-34c76f144340 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
34142024-07-25T05:48:30.173ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36519
34152024-07-25T05:48:30.174ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:673 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
34162024-07-25T05:48:30.174ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte0" rules = []
34172024-07-25T05:48:30.174ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34182024-07-25T05:48:30.174ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34192024-07-25T05:48:30.174ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
34202024-07-25T05:48:30.174ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
34212024-07-25T05:48:30.174ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34222024-07-25T05:48:30.174ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 841 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:36519 req_id = 8b8a2c18-d462-44a4-8948-9cbe7eb1d6e7 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
34232024-07-25T05:48:37.165ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50208
34242024-07-25T05:48:37.165ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 261 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50208 req_id = 02ac38d5-fb24-4524-9323-5821edc6406a response_code = 200 uri = /vpc-routes
34252024-07-25T05:48:37.230ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 2745 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:50208 req_id = 36629143-5151-43c5-bac7-427f5368d67d response_code = 204 uri = /vpc-routes
34262024-07-25T05:48:54.473ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:39689
34272024-07-25T05:48:54.474ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 110 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:39689 req_id = 4d6aad7c-3de0-4cea-95c6-490806bbd07a response_code = 200 uri = /vpc-routes
34282024-07-25T05:48:54.481ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 118 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:39689 req_id = f5d28281-d231-40e4-8800-5924bfb4b4b1 response_code = 204 uri = /vpc-routes
34292024-07-25T05:48:54.518ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36372
34302024-07-25T05:48:54.519ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 1065 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:36372 req_id = ed729cb7-7d1d-4624-bc8e-e07b5e375d82 response_code = 200 uri = /v2p
34312024-07-25T05:48:54.661ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:58226
34322024-07-25T05:48:54.662ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
34332024-07-25T05:48:54.714ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
34342024-07-25T05:48:54.746ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 84515 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:58226 req_id = 08d86f4f-8774-4655-a70a-ddfdeffa2239 response_code = 204 uri = /switch-ports
34352024-07-25T05:48:54.747ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 821 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:58226 req_id = e6d37640-30f2-414f-a6e9-8a4feca889cf response_code = 200 uri = /network-bootstore-config
34362024-07-25T05:48:54.781ZINFOSledAgent (bootstore): Attempting to update network config with generation: 2, current_generation: 1 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-buskin-i86pc
34372024-07-25T05:48:54.781ZINFOSledAgent (bootstore): Writing ledger to /pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
34382024-07-25T05:48:54.781ZINFOSledAgent (bootstore): Writing ledger to /pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
34392024-07-25T05:48:54.783ZINFOSledAgent (bootstore): Broadcasting network config with generation 2 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-buskin-i86pc
34402024-07-25T05:48:54.783ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 3511 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:58226 req_id = c3ed24c5-062b-418c-99ee-35eb61574722 response_code = 204 uri = /network-bootstore-config
34412024-07-25T05:48:54.853ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:56316
34422024-07-25T05:48:54.853ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 67 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:56316 req_id = 3729db6f-250e-4eb4-8a07-7ff4f7ba91e5 response_code = 200 uri = /vpc-routes
34432024-07-25T05:48:54.861ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 152 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:56316 req_id = c1135637-21a8-46f5-9137-d8b1352c42cc response_code = 204 uri = /vpc-routes
34442024-07-25T05:48:54.982ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:44425
34452024-07-25T05:48:54.982ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 29 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:44425 req_id = f0a49be1-aaf3-4e32-a545-fe1708e428e6 response_code = 200 uri = /baseboard
34462024-07-25T05:48:55.002ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:34075
34472024-07-25T05:48:55.002ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 133 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:34075 req_id = fc0df957-46da-4029-8f6e-b57e55bff1af response_code = 200 uri = /v2p
34482024-07-25T05:48:55.048ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63210
34492024-07-25T05:48:55.048ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
34502024-07-25T05:48:55.074ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
34512024-07-25T05:48:55.095ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 46735 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:63210 req_id = 45af84a3-97ed-47d6-9094-684d981136c8 response_code = 204 uri = /switch-ports
34522024-07-25T05:48:55.095ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 98 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:63210 req_id = a684c1d9-64b9-4402-b436-ac8b7f01e5df response_code = 200 uri = /network-bootstore-config
34532024-07-25T05:48:55.158ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:54069
34542024-07-25T05:48:55.159ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 729 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:54069 req_id = e505292e-64f8-4b80-83d5-30ebea24326e response_code = 200 uri = /vpc-routes
34552024-07-25T05:48:55.167ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 111 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:54069 req_id = 26103702-4dda-44a0-b21f-9e946cb72f6d response_code = 204 uri = /vpc-routes
34562024-07-25T05:48:55.204ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:55742
34572024-07-25T05:48:55.205ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 837 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:55742 req_id = 5b2f9e33-db37-4efc-9950-0f8933d26fb3 response_code = 200 uri = /v2p
34582024-07-25T05:48:55.228ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:54636
34592024-07-25T05:48:55.228ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
34602024-07-25T05:48:55.251ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
34612024-07-25T05:48:55.271ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 43425 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:54636 req_id = 23e1497a-69cb-4eea-aaa4-0d0f19c1e053 response_code = 204 uri = /switch-ports
34622024-07-25T05:48:55.271ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 80 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:54636 req_id = b8b655d4-301a-4311-8950-4de1a0ff1f42 response_code = 200 uri = /network-bootstore-config
34632024-07-25T05:49:15.692ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
34642024-07-25T05:49:24.481ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:48397
34652024-07-25T05:49:24.482ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 817 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:48397 req_id = c4df246a-64ce-4c1f-9a83-d3446c3d230e response_code = 200 uri = /vpc-routes
34662024-07-25T05:49:24.489ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 123 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:48397 req_id = 0b5a3b1c-6676-47c1-b71a-fee42786a78d response_code = 204 uri = /vpc-routes
34672024-07-25T05:49:24.566ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47540
34682024-07-25T05:49:24.567ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 114 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47540 req_id = 5efcdc7d-727e-4979-ae10-ad9120dc228a response_code = 200 uri = /v2p
34692024-07-25T05:49:24.897ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:51391
34702024-07-25T05:49:24.897ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
34712024-07-25T05:49:24.921ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
34722024-07-25T05:49:24.944ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 47437 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:51391 req_id = 42582f48-1d56-48fa-b6a3-529664e84c77 response_code = 204 uri = /switch-ports
34732024-07-25T05:49:24.945ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 210 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:51391 req_id = e065b31d-3665-44e1-a663-ffbacbbfe7de response_code = 200 uri = /network-bootstore-config
34742024-07-25T05:49:25.015ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:57676
34752024-07-25T05:49:25.015ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 43 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:57676 req_id = 4788cb1a-8e1b-4e62-bce4-c01a65361476 response_code = 200 uri = /baseboard
34762024-07-25T05:49:25.119ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:48565
34772024-07-25T05:49:25.119ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 38 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:48565 req_id = 8d9b7045-35e2-4fe0-b1aa-a59d68aa30a5 response_code = 200 uri = /vpc-routes
34782024-07-25T05:49:25.127ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 104 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:48565 req_id = b6c1c553-19fb-4923-bc09-1559d1329429 response_code = 204 uri = /vpc-routes
34792024-07-25T05:49:25.163ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:46355
34802024-07-25T05:49:25.164ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 1007 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:46355 req_id = e0f5301e-a742-44c0-8d92-96a61a3be012 response_code = 200 uri = /vpc-routes
34812024-07-25T05:49:25.171ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 70 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:46355 req_id = 319aa3ba-d8ec-419c-85ec-6c9ac7bd8eb7 response_code = 204 uri = /vpc-routes
34822024-07-25T05:49:25.196ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37458
34832024-07-25T05:49:25.196ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 284 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:37458 req_id = 6d6392ba-9e4e-47c8-9bf9-1cd5c851dcb1 response_code = 200 uri = /v2p
34842024-07-25T05:49:25.219ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:33594
34852024-07-25T05:49:25.219ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:33594 req_id = fa2d985d-42d1-45a1-a219-619eaa69474e response_code = 200 uri = /v2p
34862024-07-25T05:49:25.342ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:57997
34872024-07-25T05:49:25.343ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
34882024-07-25T05:49:25.370ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
34892024-07-25T05:49:25.395ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 52040 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:57997 req_id = 335b2984-a633-4e87-a404-9d7427e36903 response_code = 204 uri = /switch-ports
34902024-07-25T05:49:25.396ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 916 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:57997 req_id = c1b50b4e-939c-4599-9a59-9f43baea893f response_code = 200 uri = /network-bootstore-config
34912024-07-25T05:49:25.494ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:64881
34922024-07-25T05:49:25.494ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
34932024-07-25T05:49:25.520ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
34942024-07-25T05:49:25.545ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 50774 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:64881 req_id = 58e7261b-a7a6-4f39-92e3-aa247181e3d3 response_code = 204 uri = /switch-ports
34952024-07-25T05:49:25.545ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 84 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:64881 req_id = 81ec800f-47aa-4337-b629-2b1fa007fe71 response_code = 200 uri = /network-bootstore-config
34962024-07-25T05:49:54.350ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:56765
34972024-07-25T05:49:54.350ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 117 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:56765 req_id = f0caddde-402a-4c04-808c-5e18a4b1501f response_code = 200 uri = /vpc-routes
34982024-07-25T05:49:54.358ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 793 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:56765 req_id = c8982c67-b69e-43a8-b27f-2a1f49e408b1 response_code = 204 uri = /vpc-routes
34992024-07-25T05:49:54.439ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59998
35002024-07-25T05:49:54.439ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 144 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:59998 req_id = 48d8ab32-807e-4923-9867-37bbf633007e response_code = 200 uri = /v2p
35012024-07-25T05:49:54.538ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42335
35022024-07-25T05:49:54.538ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
35032024-07-25T05:49:54.566ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
35042024-07-25T05:49:54.588ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 49948 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:42335 req_id = 6c2931f9-58d7-40b0-819b-82ae9c787074 response_code = 204 uri = /switch-ports
35052024-07-25T05:49:54.589ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 151 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:42335 req_id = c3a890c2-a576-4ed0-8c1b-9fa9664d197e response_code = 200 uri = /network-bootstore-config
35062024-07-25T05:49:54.868ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:45679
35072024-07-25T05:49:54.868ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 108 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:45679 req_id = 1d2c0793-584e-4f29-a055-4654f27cb8ab response_code = 200 uri = /vpc-routes
35082024-07-25T05:49:54.875ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 118 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:45679 req_id = 18d4c648-4ef2-4ed0-9c22-4c9332c0cc6a response_code = 204 uri = /vpc-routes
35092024-07-25T05:49:54.911ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63074
35102024-07-25T05:49:54.911ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 121 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:63074 req_id = 088a65aa-8cd6-4aaf-bd69-987c11b199f1 response_code = 200 uri = /v2p
35112024-07-25T05:49:54.911ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:63208
35122024-07-25T05:49:54.912ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 34 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:63208 req_id = 752bc221-148d-4c03-9903-4d06ff231673 response_code = 200 uri = /vpc-routes
35132024-07-25T05:49:54.918ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 58 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:63208 req_id = 84bf5c38-7e67-40c5-a3cc-fed7ec44c91e response_code = 204 uri = /vpc-routes
35142024-07-25T05:49:55.020ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:49266
35152024-07-25T05:49:55.020ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 132 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:49266 req_id = 5af92990-d7e5-480c-bfa7-2d6def609025 response_code = 200 uri = /v2p
35162024-07-25T05:49:55.038ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:35944
35172024-07-25T05:49:55.038ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
35182024-07-25T05:49:55.062ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
35192024-07-25T05:49:55.083ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 44890 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:35944 req_id = 2eb03efa-20da-489d-86fa-ef4bc9f5846e response_code = 204 uri = /switch-ports
35202024-07-25T05:49:55.083ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:34433
35212024-07-25T05:49:55.083ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 61 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:34433 req_id = 76edc706-bc72-4a60-a454-9554e188e7f6 response_code = 200 uri = /baseboard
35222024-07-25T05:49:55.084ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 154 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:35944 req_id = 48d9b7dc-0502-4716-93ee-594aac110b1f response_code = 200 uri = /network-bootstore-config
35232024-07-25T05:49:55.113ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:52711
35242024-07-25T05:49:55.114ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
35252024-07-25T05:49:55.135ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
35262024-07-25T05:49:55.156ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 42514 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:52711 req_id = 15174613-52d7-40f3-993c-d1a2a83c9442 response_code = 204 uri = /switch-ports
35272024-07-25T05:49:55.156ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 85 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:52711 req_id = 4aa88312-3d2c-49c9-aff9-cd9bc55cbe61 response_code = 200 uri = /network-bootstore-config
35282024-07-25T05:50:13.055ZINFOSledAgent (dropshot): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:57429 remote_addr = [fd00:1122:3344:101::d]:46986 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376
35292024-07-25T05:50:13.064ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 8213 local_addr = [fd00:1122:3344:101::1]:57429 method = GET remote_addr = [fd00:1122:3344:101::d]:46986 req_id = 0dbcaf0c-c379-4655-b3f0-c06be8b85a38 response_code = 200 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376 uri = /8fa4bc78-1026-41f0-b09d-c67224071376
35302024-07-25T05:50:24.694ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:55261
35312024-07-25T05:50:24.694ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:55261 req_id = c3f168c5-5b4d-4e2f-9f46-8dd78b0f1cb9 response_code = 200 uri = /vpc-routes
35322024-07-25T05:50:24.701ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 133 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:55261 req_id = edf4b9ea-4a2c-477e-afaf-e5ec46de4965 response_code = 204 uri = /vpc-routes
35332024-07-25T05:50:24.760ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61931
35342024-07-25T05:50:24.762ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 1896 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:61931 req_id = 0df04db0-ce03-4261-9cc2-e5132603247e response_code = 200 uri = /v2p
35352024-07-25T05:50:24.881ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61883
35362024-07-25T05:50:24.881ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
35372024-07-25T05:50:24.911ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
35382024-07-25T05:50:24.940ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 58525 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:61883 req_id = 8a259190-82fd-42c5-85a7-8e0dfbc0514a response_code = 204 uri = /switch-ports
35392024-07-25T05:50:24.940ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:61883 req_id = e5769390-6a41-4643-aac9-ea8e29ccdfca response_code = 200 uri = /network-bootstore-config
35402024-07-25T05:50:25.020ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:47161
35412024-07-25T05:50:25.020ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 116 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:47161 req_id = 93e3afa7-07d0-4b7b-be13-4d213807b305 response_code = 200 uri = /v2p
35422024-07-25T05:50:25.063ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:58235
35432024-07-25T05:50:25.063ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 80 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:58235 req_id = d4b6eb3e-25de-4468-8d88-61f89329aa25 response_code = 200 uri = /v2p
35442024-07-25T05:50:25.065ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:36964
35452024-07-25T05:50:25.066ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 905 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:36964 req_id = 33f10fbf-19c7-4232-81c5-11e076454d1f response_code = 200 uri = /vpc-routes
35462024-07-25T05:50:25.074ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 95 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:36964 req_id = 45c90186-277a-4f2e-a262-4e345f2a7b77 response_code = 204 uri = /vpc-routes
35472024-07-25T05:50:25.117ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:60989
35482024-07-25T05:50:25.118ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 34 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:60989 req_id = cedb7a5d-574c-4c7d-a28c-dd9c6880dfa2 response_code = 200 uri = /vpc-routes
35492024-07-25T05:50:25.121ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:39459
35502024-07-25T05:50:25.122ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 519 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:39459 req_id = 8f73be18-b4f4-4026-a7c8-f2a1bfdfb555 response_code = 200 uri = /baseboard
35512024-07-25T05:50:25.125ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 78 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:60989 req_id = 279a65bd-a44d-421e-a8ec-522974b99329 response_code = 204 uri = /vpc-routes
35522024-07-25T05:50:25.269ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:45209
35532024-07-25T05:50:25.269ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
35542024-07-25T05:50:25.294ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
35552024-07-25T05:50:25.318ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 48846 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:45209 req_id = 834d3142-399d-4128-9a64-ba1120369888 response_code = 204 uri = /switch-ports
35562024-07-25T05:50:25.318ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:60019
35572024-07-25T05:50:25.318ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
35582024-07-25T05:50:25.343ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
35592024-07-25T05:50:25.365ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 46875 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:60019 req_id = 554d9f51-6aa0-451d-a2a0-3e8b6426f3c1 response_code = 204 uri = /switch-ports
35602024-07-25T05:50:25.365ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 138 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:45209 req_id = b723fb7f-bc22-4a70-8f03-350a31b763ee response_code = 200 uri = /network-bootstore-config
35612024-07-25T05:50:25.366ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 629 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:60019 req_id = 3abf891f-bce2-4195-ab16-7d70f0a1712e response_code = 200 uri = /network-bootstore-config
35622024-07-25T05:50:43.055ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 876 local_addr = [fd00:1122:3344:101::1]:57429 method = GET remote_addr = [fd00:1122:3344:101::d]:46986 req_id = 4db7a64e-505e-400c-8165-5ca67e50b8b9 response_code = 200 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376 uri = /8fa4bc78-1026-41f0-b09d-c67224071376
35632024-07-25T05:50:54.385ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:46311
35642024-07-25T05:50:54.385ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 98 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:46311 req_id = 8b37d6e2-7f04-456d-b41e-c04d515f9c16 response_code = 200 uri = /vpc-routes
35652024-07-25T05:50:54.391ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 123 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:46311 req_id = 400191f8-745d-47d2-b34b-90771b578a81 response_code = 204 uri = /vpc-routes
35662024-07-25T05:50:54.503ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:45342
35672024-07-25T05:50:54.503ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 146 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:45342 req_id = 551e6f4d-52ea-49a6-b0c3-8e92dd9b92a4 response_code = 200 uri = /v2p
35682024-07-25T05:50:54.694ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34741
35692024-07-25T05:50:54.694ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
35702024-07-25T05:50:54.719ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
35712024-07-25T05:50:54.746ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 52530 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:34741 req_id = 9cf65a7c-44d4-4e00-a249-e8200b4f2450 response_code = 204 uri = /switch-ports
35722024-07-25T05:50:54.747ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 100 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:34741 req_id = aabd8317-dcdb-4a09-b455-183103600b86 response_code = 200 uri = /network-bootstore-config
35732024-07-25T05:50:54.933ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:55582
35742024-07-25T05:50:54.933ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 168 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:55582 req_id = 56c44973-3848-4317-95f7-5665a8e2453b response_code = 200 uri = /v2p
35752024-07-25T05:50:54.939ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40475
35762024-07-25T05:50:54.939ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 55 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40475 req_id = 233779a1-2033-4c09-971a-6d41cdbd4266 response_code = 200 uri = /vpc-routes
35772024-07-25T05:50:54.946ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 93 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:40475 req_id = 0eaf7b10-667e-4dc0-9353-57c0b85c399c response_code = 204 uri = /vpc-routes
35782024-07-25T05:50:54.985ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:52131
35792024-07-25T05:50:54.985ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 56 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:52131 req_id = 87efa877-2567-4edf-ba5b-fe6b0af6f2e7 response_code = 200 uri = /vpc-routes
35802024-07-25T05:50:54.992ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 709 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:52131 req_id = 43cc8c62-b54d-4e7f-a88c-eae5ed77f545 response_code = 204 uri = /vpc-routes
35812024-07-25T05:50:55.000ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:53589
35822024-07-25T05:50:55.000ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 95 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:53589 req_id = 42fd33d9-7fbb-433f-a7dd-9e8899d67699 response_code = 200 uri = /v2p
35832024-07-25T05:50:55.071ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:64054
35842024-07-25T05:50:55.071ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
35852024-07-25T05:50:55.098ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
35862024-07-25T05:50:55.121ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 49720 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:64054 req_id = d4e980a6-6ddf-4ba4-8c07-375d686cd633 response_code = 204 uri = /switch-ports
35872024-07-25T05:50:55.121ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:64347
35882024-07-25T05:50:55.121ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 112 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:64054 req_id = 8060904a-6329-42ab-8b07-38eb0de96d9d response_code = 200 uri = /network-bootstore-config
35892024-07-25T05:50:55.121ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
35902024-07-25T05:50:55.147ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
35912024-07-25T05:50:55.155ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:50595
35922024-07-25T05:50:55.155ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 86 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:50595 req_id = 53d1819c-995a-4cde-9b94-5499503bd698 response_code = 200 uri = /baseboard
35932024-07-25T05:50:55.169ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 47977 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:64347 req_id = dc0cbbe9-a791-4e06-bbd7-3e270c30788d response_code = 204 uri = /switch-ports
35942024-07-25T05:50:55.170ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 125 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:64347 req_id = 9a7a6078-3cee-4acf-a0d4-255659df6316 response_code = 200 uri = /network-bootstore-config
35952024-07-25T05:51:13.055ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 572 local_addr = [fd00:1122:3344:101::1]:57429 method = GET remote_addr = [fd00:1122:3344:101::d]:46986 req_id = a0d5f363-c958-4879-8831-fae3eb6b4152 response_code = 200 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376 uri = /8fa4bc78-1026-41f0-b09d-c67224071376
35962024-07-25T05:51:24.481ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:37476
35972024-07-25T05:51:24.481ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 206 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:37476 req_id = 20b70580-b139-42e6-a01d-3d5fe4e1579b response_code = 200 uri = /v2p
35982024-07-25T05:51:24.627ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42944
35992024-07-25T05:51:24.627ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 48 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:42944 req_id = 3cb0c9f1-e2a3-43a2-af58-205bc6d28cb2 response_code = 200 uri = /vpc-routes
36002024-07-25T05:51:24.635ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 111 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:42944 req_id = 5a538f53-1bfc-4fb3-b26d-e8e2d6c1a68d response_code = 204 uri = /vpc-routes
36012024-07-25T05:51:24.935ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:46747
36022024-07-25T05:51:24.936ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
36032024-07-25T05:51:24.962ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
36042024-07-25T05:51:24.985ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 50391 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:46747 req_id = f2a2c44c-6e19-440b-bbda-3e81cc5cdaaa response_code = 204 uri = /switch-ports
36052024-07-25T05:51:24.986ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 115 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:46747 req_id = 102a6627-eda4-4e6a-a380-627a77caed09 response_code = 200 uri = /network-bootstore-config
36062024-07-25T05:51:25.172ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:53364
36072024-07-25T05:51:25.173ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 144 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:53364 req_id = b45f15a3-caaf-4281-9410-4024b0932ae7 response_code = 200 uri = /v2p
36082024-07-25T05:51:25.174ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:32952
36092024-07-25T05:51:25.175ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 30 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:32952 req_id = 695e02a5-3bd5-43d0-8229-17bd4c138ad2 response_code = 200 uri = /vpc-routes
36102024-07-25T05:51:25.178ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:49745
36112024-07-25T05:51:25.178ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 79 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:49745 req_id = 2a8d442d-7057-4832-9f40-7d47189dcf45 response_code = 200 uri = /v2p
36122024-07-25T05:51:25.183ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 83 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:32952 req_id = 1796e1d4-41a0-4fbe-a9a7-f50347daec02 response_code = 204 uri = /vpc-routes
36132024-07-25T05:51:25.190ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:52732
36142024-07-25T05:51:25.190ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 27 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:52732 req_id = 155cd9e1-6f41-4f19-89fd-c5c0b8b4082c response_code = 200 uri = /baseboard
36152024-07-25T05:51:25.286ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:50282
36162024-07-25T05:51:25.286ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 47 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:50282 req_id = 86fe111e-ed02-4ba2-8968-aa97fefe8125 response_code = 200 uri = /vpc-routes
36172024-07-25T05:51:25.295ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 862 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:50282 req_id = ceffa0fe-1ddb-44a1-aba6-aee746492c5e response_code = 204 uri = /vpc-routes
36182024-07-25T05:51:25.337ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:57404
36192024-07-25T05:51:25.337ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
36202024-07-25T05:51:25.363ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
36212024-07-25T05:51:25.387ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 49838 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:57404 req_id = 965ef44d-7bd1-4058-aec7-06be58808d2c response_code = 204 uri = /switch-ports
36222024-07-25T05:51:25.387ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 85 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:57404 req_id = 207cd807-8e17-4111-97fa-29be1d9b5ec5 response_code = 200 uri = /network-bootstore-config
36232024-07-25T05:51:25.477ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:52796
36242024-07-25T05:51:25.477ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
36252024-07-25T05:51:25.500ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
36262024-07-25T05:51:25.523ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 45477 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:52796 req_id = cb5b14c4-c507-4418-bb51-9f5edaf02af8 response_code = 204 uri = /switch-ports
36272024-07-25T05:51:25.523ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 66 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:52796 req_id = a69c5a91-bb8f-4c52-ab40-8ebfa7769531 response_code = 200 uri = /network-bootstore-config
36282024-07-25T05:51:43.055ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 854 local_addr = [fd00:1122:3344:101::1]:57429 method = GET remote_addr = [fd00:1122:3344:101::d]:46986 req_id = f24b5984-10aa-4ca5-a9ce-1dcfe993e687 response_code = 200 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376 uri = /8fa4bc78-1026-41f0-b09d-c67224071376
36292024-07-25T05:51:54.412ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:60944
36302024-07-25T05:51:54.412ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 108 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:60944 req_id = 1f1c22c3-b293-4f6d-9606-af42b33e5eb0 response_code = 200 uri = /vpc-routes
36312024-07-25T05:51:54.419ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 123 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:60944 req_id = 9dcbdeef-b26e-4a41-8b8a-16052a232c50 response_code = 204 uri = /vpc-routes
36322024-07-25T05:51:54.507ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57271
36332024-07-25T05:51:54.508ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 168 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:57271 req_id = 8c866a1d-2611-4f2d-b50b-e0063d96c136 response_code = 200 uri = /v2p
36342024-07-25T05:51:54.633ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57565
36352024-07-25T05:51:54.633ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
36362024-07-25T05:51:54.659ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
36372024-07-25T05:51:54.684ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 51037 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:57565 req_id = 9f921910-fb92-4a8d-80cb-7cce7c7b6f51 response_code = 204 uri = /switch-ports
36382024-07-25T05:51:54.685ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 724 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:57565 req_id = 3e15e7cb-1108-48ce-921f-8b20fc81a350 response_code = 200 uri = /network-bootstore-config
36392024-07-25T05:51:54.888ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:34749
36402024-07-25T05:51:54.888ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 110 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:34749 req_id = fc4be165-3c0d-42a1-a9d9-5193a81a1237 response_code = 200 uri = /vpc-routes
36412024-07-25T05:51:54.894ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:46493
36422024-07-25T05:51:54.894ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 57 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:46493 req_id = a99ee55e-28fe-4428-9566-be8e175f0c5d response_code = 200 uri = /vpc-routes
36432024-07-25T05:51:54.895ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 130 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:34749 req_id = e55456e7-3c68-4e98-9848-7f5ca52c09cd response_code = 204 uri = /vpc-routes
36442024-07-25T05:51:54.902ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 792 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:46493 req_id = b854c055-139d-454b-a115-8a1f6147561d response_code = 204 uri = /vpc-routes
36452024-07-25T05:51:54.965ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61748
36462024-07-25T05:51:54.966ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 135 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61748 req_id = 15b9be5f-3814-4819-a2fc-10efee943793 response_code = 200 uri = /v2p
36472024-07-25T05:51:54.978ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:43589
36482024-07-25T05:51:54.978ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 103 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:43589 req_id = 149a1511-2fc6-474e-a15f-5fa1f48f8549 response_code = 200 uri = /v2p
36492024-07-25T05:51:55.075ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:35983
36502024-07-25T05:51:55.075ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
36512024-07-25T05:51:55.102ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
36522024-07-25T05:51:55.124ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 48816 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:35983 req_id = 63822523-1187-43f7-a6f0-6ab4cc72df68 response_code = 204 uri = /switch-ports
36532024-07-25T05:51:55.124ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:54176
36542024-07-25T05:51:55.125ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
36552024-07-25T05:51:55.130ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 5480 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:35983 req_id = 67036130-a74e-4e5c-a904-36a54225370d response_code = 200 uri = /network-bootstore-config
36562024-07-25T05:51:55.150ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
36572024-07-25T05:51:55.171ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 47030 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:54176 req_id = d790ed48-d99d-4925-89dc-7eda38151522 response_code = 204 uri = /switch-ports
36582024-07-25T05:51:55.172ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 114 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:54176 req_id = 3ae23da7-3d24-4ab8-8577-fe7dff2011a8 response_code = 200 uri = /network-bootstore-config
36592024-07-25T05:51:55.223ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:59794
36602024-07-25T05:51:55.224ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 44 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:59794 req_id = 90839fa5-81af-46ba-882a-5af129f43e07 response_code = 200 uri = /baseboard
36612024-07-25T05:52:13.063ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 295 local_addr = [fd00:1122:3344:101::1]:57429 method = GET remote_addr = [fd00:1122:3344:101::d]:46986 req_id = e7d26e23-1e7d-4667-9243-d3250391ced3 response_code = 200 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376 uri = /8fa4bc78-1026-41f0-b09d-c67224071376
36622024-07-25T05:52:24.459ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52242
36632024-07-25T05:52:24.459ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 119 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:52242 req_id = c96e8857-e351-4e95-8fa3-d38f5285f335 response_code = 200 uri = /vpc-routes
36642024-07-25T05:52:24.466ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 132 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:52242 req_id = f9fcad9d-3e95-4e43-abfa-7ca571a0d1b6 response_code = 204 uri = /vpc-routes
36652024-07-25T05:52:24.508ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:49272
36662024-07-25T05:52:24.509ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 116 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:49272 req_id = e9b493bd-d4db-411d-a340-3273f7ccf5d8 response_code = 200 uri = /v2p
36672024-07-25T05:52:24.812ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:38869
36682024-07-25T05:52:24.812ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
36692024-07-25T05:52:24.840ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
36702024-07-25T05:52:24.864ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 52490 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:38869 req_id = faf9e2ca-a68f-4323-9f6e-ed3635f6dc7b response_code = 204 uri = /switch-ports
36712024-07-25T05:52:24.865ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 106 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:38869 req_id = 51793651-9992-441b-be23-0322388aef88 response_code = 200 uri = /network-bootstore-config
36722024-07-25T05:52:24.935ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50231
36732024-07-25T05:52:24.935ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 73 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50231 req_id = 437b1c00-aba4-4bb1-9844-e34f3fdff33f response_code = 200 uri = /vpc-routes
36742024-07-25T05:52:24.943ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 757 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:50231 req_id = 3724562c-dd55-4296-9392-e9dcdabe4c3c response_code = 204 uri = /vpc-routes
36752024-07-25T05:52:24.969ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:42393
36762024-07-25T05:52:24.969ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 106 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:42393 req_id = b8e76206-8523-40d2-88f7-1728d1a621b4 response_code = 200 uri = /v2p
36772024-07-25T05:52:25.219ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:63017
36782024-07-25T05:52:25.219ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 33 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:63017 req_id = d970257e-7c88-4f00-99cb-97f075dbdaf1 response_code = 200 uri = /vpc-routes
36792024-07-25T05:52:25.228ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 1053 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:63017 req_id = e9dc0de2-858a-4003-930e-26e2425dc4df response_code = 204 uri = /vpc-routes
36802024-07-25T05:52:25.242ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:35756
36812024-07-25T05:52:25.243ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 103 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:35756 req_id = 56415777-46a0-4d2e-9c7c-1cbb41b84f15 response_code = 200 uri = /v2p
36822024-07-25T05:52:25.255ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:48361
36832024-07-25T05:52:25.255ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 28 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:48361 req_id = daa22b88-54b1-4124-b327-6ff7baa8e189 response_code = 200 uri = /baseboard
36842024-07-25T05:52:25.405ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50384
36852024-07-25T05:52:25.405ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
36862024-07-25T05:52:25.432ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
36872024-07-25T05:52:25.456ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 51352 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:50384 req_id = ca45c504-57cb-4338-8401-e477c0b8a70a response_code = 204 uri = /switch-ports
36882024-07-25T05:52:25.456ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:35708
36892024-07-25T05:52:25.456ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 108 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50384 req_id = b996e44e-2934-46f9-acad-3477ddc4c6df response_code = 200 uri = /network-bootstore-config
36902024-07-25T05:52:25.456ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
36912024-07-25T05:52:25.482ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
36922024-07-25T05:52:25.505ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 48293 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:35708 req_id = ee095070-0e13-4e27-84f9-df1aed12ffbb response_code = 204 uri = /switch-ports
36932024-07-25T05:52:25.505ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 108 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:35708 req_id = f0db29b8-fe21-4d74-ab5e-84aadec9a86e response_code = 200 uri = /network-bootstore-config
36942024-07-25T05:52:43.104ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 569 local_addr = [fd00:1122:3344:101::1]:57429 method = GET remote_addr = [fd00:1122:3344:101::d]:46986 req_id = 0d0b7ff4-c485-421d-acf7-ad866c210521 response_code = 200 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376 uri = /8fa4bc78-1026-41f0-b09d-c67224071376
36952024-07-25T05:52:54.412ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33169
36962024-07-25T05:52:54.412ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 92 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:33169 req_id = 2e6efd11-7fca-4dd1-a19d-7315aecbbda9 response_code = 200 uri = /vpc-routes
36972024-07-25T05:52:54.419ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 93 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:33169 req_id = b7195479-cd67-4d06-ac70-a66b55c2bb78 response_code = 204 uri = /vpc-routes
36982024-07-25T05:52:54.553ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59647
36992024-07-25T05:52:54.553ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 173 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:59647 req_id = 70b08669-7008-40f2-9e93-e1095f9ad0c8 response_code = 200 uri = /v2p
37002024-07-25T05:52:54.612ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:48284
37012024-07-25T05:52:54.613ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
37022024-07-25T05:52:54.639ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
37032024-07-25T05:52:54.662ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 50023 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:48284 req_id = a49b39bb-bf82-4c04-97ef-af68ac0ff3e7 response_code = 204 uri = /switch-ports
37042024-07-25T05:52:54.662ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 94 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:48284 req_id = 680e69ba-86cf-4a3e-ba78-4f674b57021d response_code = 200 uri = /network-bootstore-config
37052024-07-25T05:52:54.882ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:57067
37062024-07-25T05:52:54.882ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 110 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:57067 req_id = 0c620675-f0f1-49ca-b467-7d47ca846c5b response_code = 200 uri = /vpc-routes
37072024-07-25T05:52:54.890ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 137 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:57067 req_id = 5c2f6c4c-4df7-41a4-858c-e12868a23b33 response_code = 204 uri = /vpc-routes
37082024-07-25T05:52:54.901ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:47953
37092024-07-25T05:52:54.902ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 970 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:47953 req_id = 7f2d7006-7964-452a-b5cd-342025084ea3 response_code = 200 uri = /v2p
37102024-07-25T05:52:54.951ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44737
37112024-07-25T05:52:54.951ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 41 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44737 req_id = ae1a1570-3a7c-4b47-bc71-325d78e96acb response_code = 200 uri = /vpc-routes
37122024-07-25T05:52:54.960ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 80 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:44737 req_id = 20d31239-1aa4-4157-bb51-e3a81c69ea03 response_code = 204 uri = /vpc-routes
37132024-07-25T05:52:55.030ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:46958
37142024-07-25T05:52:55.031ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 838 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:46958 req_id = 535254e1-4f09-4ac2-97be-69ebdeddfcdc response_code = 200 uri = /v2p
37152024-07-25T05:52:55.090ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50297
37162024-07-25T05:52:55.090ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
37172024-07-25T05:52:55.115ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
37182024-07-25T05:52:55.139ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 49414 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:50297 req_id = 25a0b27c-c267-4b1c-af1a-6194bfa7d8c1 response_code = 204 uri = /switch-ports
37192024-07-25T05:52:55.139ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:39281
37202024-07-25T05:52:55.141ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
37212024-07-25T05:52:55.164ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
37222024-07-25T05:52:55.185ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 45518 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:39281 req_id = 49975b0e-c2ba-4451-93c2-299f7f7370a4 response_code = 204 uri = /switch-ports
37232024-07-25T05:52:55.186ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 179 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50297 req_id = a2f7ec23-3b5d-400f-ad5f-167a04d96775 response_code = 200 uri = /network-bootstore-config
37242024-07-25T05:52:55.186ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 146 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:39281 req_id = 075f6059-a15d-4b12-83bd-18022641d15f response_code = 200 uri = /network-bootstore-config
37252024-07-25T05:52:55.289ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:43888
37262024-07-25T05:52:55.289ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 65 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:43888 req_id = 945015b9-2b6a-443b-96dd-06d1892e3494 response_code = 200 uri = /baseboard
37272024-07-25T05:53:13.106ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 2410 local_addr = [fd00:1122:3344:101::1]:57429 method = GET remote_addr = [fd00:1122:3344:101::d]:46986 req_id = fcd85374-cebd-4fea-a747-ac2ac2c64c52 response_code = 200 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376 uri = /8fa4bc78-1026-41f0-b09d-c67224071376
37282024-07-25T05:53:24.506ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:44990
37292024-07-25T05:53:24.507ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 933 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:44990 req_id = 03883dee-6218-4d4f-93c7-7d521802cfd5 response_code = 200 uri = /v2p
37302024-07-25T05:53:24.666ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63644
37312024-07-25T05:53:24.666ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:673 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
37322024-07-25T05:53:24.666ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte0" rules = []
37332024-07-25T05:53:24.666ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
37342024-07-25T05:53:24.666ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
37352024-07-25T05:53:24.666ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
37362024-07-25T05:53:24.666ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
37372024-07-25T05:53:24.666ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
37382024-07-25T05:53:24.666ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 480 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:63644 req_id = 27c16381-f3c2-4e8e-9e81-3ee6204fcf72 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
37392024-07-25T05:53:24.698ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62627
37402024-07-25T05:53:24.698ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 38 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:62627 req_id = 02c98f30-2b1f-49d7-93a2-38a6b2c15861 response_code = 200 uri = /vpc-routes
37412024-07-25T05:53:24.705ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 79 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:62627 req_id = cdbd03f9-2fa7-49ee-b542-ba0beef7605a response_code = 204 uri = /vpc-routes
37422024-07-25T05:53:24.872ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33141
37432024-07-25T05:53:24.872ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
37442024-07-25T05:53:24.901ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
37452024-07-25T05:53:24.922ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 50516 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:33141 req_id = af6c66ab-32e1-4fc6-8f9d-aa0fb7aa428d response_code = 204 uri = /switch-ports
37462024-07-25T05:53:24.924ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 1273 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:33141 req_id = 17847ee7-9aa8-41e1-9ec7-c10daeaaccd7 response_code = 200 uri = /network-bootstore-config
37472024-07-25T05:53:24.993ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63442
37482024-07-25T05:53:24.993ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 210 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:63442 req_id = 979d09ba-a2e9-4a9c-8637-aff4871a46de response_code = 200 uri = /v2p
37492024-07-25T05:53:25.009ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:43533
37502024-07-25T05:53:25.009ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 42 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:43533 req_id = 32ca88d5-f6dc-46bc-a5c8-cb5cc881cfc8 response_code = 200 uri = /vpc-routes
37512024-07-25T05:53:25.016ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 110 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:43533 req_id = 54839fe0-178e-41bb-ad03-615edb66b842 response_code = 204 uri = /vpc-routes
37522024-07-25T05:53:25.175ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:45384
37532024-07-25T05:53:25.175ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 26 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:45384 req_id = e886a522-5db9-448a-9ece-467eeabdb90a response_code = 200 uri = /vpc-routes
37542024-07-25T05:53:25.183ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 853 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:45384 req_id = 518acf99-53c1-4b1f-bd6b-6682b56bd792 response_code = 204 uri = /vpc-routes
37552024-07-25T05:53:25.213ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:47692
37562024-07-25T05:53:25.213ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:673 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
37572024-07-25T05:53:25.213ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte0" rules = []
37582024-07-25T05:53:25.213ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
37592024-07-25T05:53:25.213ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
37602024-07-25T05:53:25.213ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
37612024-07-25T05:53:25.213ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
37622024-07-25T05:53:25.213ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
37632024-07-25T05:53:25.213ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 285 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:47692 req_id = ca2d3949-a760-473b-ace0-8c1e324efc0f response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
37642024-07-25T05:53:25.244ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:65472
37652024-07-25T05:53:25.244ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 121 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:65472 req_id = 90f459fd-246b-495a-aefa-8202520b1953 response_code = 200 uri = /v2p
37662024-07-25T05:53:25.262ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:47977
37672024-07-25T05:53:25.262ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
37682024-07-25T05:53:25.286ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
37692024-07-25T05:53:25.307ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 44815 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:47977 req_id = cedc10aa-d7cf-4703-b930-c25f6d75ed97 response_code = 204 uri = /switch-ports
37702024-07-25T05:53:25.307ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 104 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:47977 req_id = b6f0d70c-82d8-4205-a6d7-d6dbd2adb8fc response_code = 200 uri = /network-bootstore-config
37712024-07-25T05:53:25.327ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:61702
37722024-07-25T05:53:25.328ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 622 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:61702 req_id = 098fb54e-2b97-41a0-a333-03f297fef238 response_code = 200 uri = /baseboard
37732024-07-25T05:53:25.417ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37281
37742024-07-25T05:53:25.418ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:673 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: a6846ef5-8686-4c7e-ae28-d178a4029a85, kind: Service { id: 2c3f8453-233f-428f-93cc-0bfefb84929d }, name: Name("external-dns-2c3f8453-233f-428f-93cc-0bfefb84929d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 164, 190])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f04fae05-ec27-43ac-ad4d-ab63c091bf1b, kind: Service { id: b04db43b-dc24-43ce-9d97-32193f88e046 }, name: Name("external-dns-b04db43b-dc24-43ce-9d97-32193f88e046"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 194, 154])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 735e2401-2ba5-4079-b4a0-7f0184716575, kind: Service { id: e4efbc31-e513-4290-82f5-c51d8afea0c7 }, name: Name("nexus-e4efbc31-e513-4290-82f5-c51d8afea0c7"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 154])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 9e8859b7-3d6e-4066-b58c-24fc535c1122, kind: Service { id: 221c5e7c-0832-4890-8b0a-f9ba08725876 }, name: Name("nexus-221c5e7c-0832-4890-8b0a-f9ba08725876"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 245, 155])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: f33554f0-0c2d-4b66-98d0-73365d475c4b, kind: Service { id: 85472347-dfd9-425e-bd98-9b81ebdf3b2b }, name: Name("nexus-85472347-dfd9-425e-bd98-9b81ebdf3b2b"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 150, 244])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
37752024-07-25T05:53:25.418ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte0" rules = []
37762024-07-25T05:53:25.418ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
37772024-07-25T05:53:25.418ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
37782024-07-25T05:53:25.418ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
37792024-07-25T05:53:25.418ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
37802024-07-25T05:53:25.418ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
37812024-07-25T05:53:25.418ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 362 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:37281 req_id = 7125b5ad-d795-45b5-82b3-d1a09c1af32c response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
37822024-07-25T05:53:25.488ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:62767
37832024-07-25T05:53:25.488ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
37842024-07-25T05:53:25.512ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
37852024-07-25T05:53:25.535ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 46752 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:62767 req_id = a79409b0-5c02-4985-a54a-053bf55e5a20 response_code = 204 uri = /switch-ports
37862024-07-25T05:53:25.536ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 745 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:62767 req_id = d92f3214-c0f8-4261-9f5b-9ad98c27e709 response_code = 200 uri = /network-bootstore-config
37872024-07-25T05:53:43.106ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 2256 local_addr = [fd00:1122:3344:101::1]:57429 method = GET remote_addr = [fd00:1122:3344:101::d]:46986 req_id = 4ed7b1a9-d119-4563-a1cf-484cfae56783 response_code = 200 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376 uri = /8fa4bc78-1026-41f0-b09d-c67224071376
37882024-07-25T05:53:54.522ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33607
37892024-07-25T05:53:54.522ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 100 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:33607 req_id = 86d6df89-d3f6-4a1d-acb4-be177928eda3 response_code = 200 uri = /vpc-routes
37902024-07-25T05:53:54.529ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 128 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:33607 req_id = e9711708-3c0f-4cb3-80ea-0744635d06f3 response_code = 204 uri = /vpc-routes
37912024-07-25T05:53:54.532ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54998
37922024-07-25T05:53:54.532ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 101 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54998 req_id = a6a49b74-3598-4926-a08f-086a443e606b response_code = 200 uri = /v2p
37932024-07-25T05:53:54.658ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:58549
37942024-07-25T05:53:54.658ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
37952024-07-25T05:53:54.682ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
37962024-07-25T05:53:54.704ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 45879 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:58549 req_id = b66ba3f9-43b7-4452-9d47-72723b7a3113 response_code = 204 uri = /switch-ports
37972024-07-25T05:53:54.705ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 646 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:58549 req_id = c279c803-6102-4f1a-ae65-3f2453eb6b63 response_code = 200 uri = /network-bootstore-config
37982024-07-25T05:53:54.871ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:56750
37992024-07-25T05:53:54.871ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 94 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:56750 req_id = 4821c2b5-2983-446d-a12d-ae0278be15b4 response_code = 200 uri = /vpc-routes
38002024-07-25T05:53:54.877ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 135 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:56750 req_id = d80fcd83-c0a8-4f10-8ea1-1d99eb496013 response_code = 204 uri = /vpc-routes
38012024-07-25T05:53:54.910ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:46483
38022024-07-25T05:53:54.910ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 44 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:46483 req_id = 1df3f34e-6efe-45ea-95a8-84dec249cef2 response_code = 200 uri = /vpc-routes
38032024-07-25T05:53:54.917ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 90 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:46483 req_id = 5b89a462-7309-4db2-81e1-32078e5bc38f response_code = 204 uri = /vpc-routes
38042024-07-25T05:53:54.970ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:45063
38052024-07-25T05:53:54.970ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 128 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:45063 req_id = f179eb8b-bb9f-42e3-b3d9-d37ca15181e7 response_code = 200 uri = /v2p
38062024-07-25T05:53:55.082ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:54018
38072024-07-25T05:53:55.082ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 95 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:54018 req_id = 28f944dd-85b7-4e38-95f2-cc4c30e99b79 response_code = 200 uri = /v2p
38082024-07-25T05:53:55.129ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:43364
38092024-07-25T05:53:55.129ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
38102024-07-25T05:53:55.154ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
38112024-07-25T05:53:55.176ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 46920 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:43364 req_id = 7322b6c5-0a24-4cb6-b367-6bde9dff1fb4 response_code = 204 uri = /switch-ports
38122024-07-25T05:53:55.176ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:51513
38132024-07-25T05:53:55.176ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
38142024-07-25T05:53:55.180ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 3393 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:43364 req_id = d5767862-466c-4adc-828e-5d588b9d62bf response_code = 200 uri = /network-bootstore-config
38152024-07-25T05:53:55.200ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
38162024-07-25T05:53:55.221ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 44968 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:51513 req_id = 5a623333-5801-419e-907f-dc2b2f4b9d3e response_code = 204 uri = /switch-ports
38172024-07-25T05:53:55.222ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 587 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:51513 req_id = 439cd6b6-68ce-4b37-ba30-789c632f71ea response_code = 200 uri = /network-bootstore-config
38182024-07-25T05:53:55.362ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:44818
38192024-07-25T05:53:55.363ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 468 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:44818 req_id = 9b51407e-4f1f-4479-96fb-6c1100f410fe response_code = 200 uri = /baseboard
38202024-07-25T05:54:01.848ZINFOSledAgent (auto-cleanup-task): running automatic periodic zone bundle cleanup file = sled-agent/src/zone_bundle.rs:303
38212024-07-25T05:54:01.848ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
38222024-07-25T05:54:01.884ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:292 next_cleanup = Instant { tv_sec: 6804, tv_nsec: 917992695 } time_to_next_cleanup = 599.99999628s
38232024-07-25T05:54:13.106ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 1655 local_addr = [fd00:1122:3344:101::1]:57429 method = GET remote_addr = [fd00:1122:3344:101::d]:46986 req_id = 246c1840-60a5-4722-86a1-fe4018aa6d3d response_code = 200 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376 uri = /8fa4bc78-1026-41f0-b09d-c67224071376
38242024-07-25T05:54:15.694ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
38252024-07-25T05:54:24.473ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:53741
38262024-07-25T05:54:24.473ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 203 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:53741 req_id = d57c38e4-7925-4031-af48-a7c65a24f809 response_code = 200 uri = /v2p
38272024-07-25T05:54:24.492ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:51483
38282024-07-25T05:54:24.492ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 700 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:51483 req_id = bce7018b-4022-46c1-962e-c494477aeda7 response_code = 200 uri = /vpc-routes
38292024-07-25T05:54:24.500ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 109 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:51483 req_id = 805d3537-9ae5-4fc5-97b2-a7fac3143244 response_code = 204 uri = /vpc-routes
38302024-07-25T05:54:24.875ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61066
38312024-07-25T05:54:24.875ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
38322024-07-25T05:54:24.906ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
38332024-07-25T05:54:24.932ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 57241 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:61066 req_id = 74ec9cd3-2b0c-4220-8350-1c9799997534 response_code = 204 uri = /switch-ports
38342024-07-25T05:54:24.938ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 101 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:61066 req_id = 59f79cf1-0708-4bc4-a777-1ab6466b88d3 response_code = 200 uri = /network-bootstore-config
38352024-07-25T05:54:24.982ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:43114
38362024-07-25T05:54:24.982ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 185 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:43114 req_id = 8cf7d59e-c974-4575-b871-86481ce6ca3a response_code = 200 uri = /v2p
38372024-07-25T05:54:25.083ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:51756
38382024-07-25T05:54:25.083ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 48 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:51756 req_id = ba9a1915-89c3-4648-aef7-dee76ad53574 response_code = 200 uri = /vpc-routes
38392024-07-25T05:54:25.091ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 110 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:51756 req_id = bb942a3c-df96-4f11-ac7c-01eae461d709 response_code = 204 uri = /vpc-routes
38402024-07-25T05:54:25.227ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58834
38412024-07-25T05:54:25.228ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 929 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58834 req_id = fd6552b2-aa0b-4cd9-988a-8419d5d16c27 response_code = 200 uri = /v2p
38422024-07-25T05:54:25.276ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:55362
38432024-07-25T05:54:25.276ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:55362 req_id = bd5ee10e-62a0-48e1-b4c0-fff04d515da3 response_code = 200 uri = /vpc-routes
38442024-07-25T05:54:25.285ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 840 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:55362 req_id = bbd538da-3a96-4fcd-a95a-3f7214c78c38 response_code = 204 uri = /vpc-routes
38452024-07-25T05:54:25.392ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:45776
38462024-07-25T05:54:25.392ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
38472024-07-25T05:54:25.426ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
38482024-07-25T05:54:25.455ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 62488 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:45776 req_id = 657462c6-6f0a-425b-b397-ee6da91c7df0 response_code = 204 uri = /switch-ports
38492024-07-25T05:54:25.455ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:65253
38502024-07-25T05:54:25.455ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:33948
38512024-07-25T05:54:25.455ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
38522024-07-25T05:54:25.459ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 4181 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:33948 req_id = d2ad83ea-3a16-40d8-8c4a-541f86aa7849 response_code = 200 uri = /baseboard
38532024-07-25T05:54:25.460ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 4439 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:45776 req_id = b051bfc7-67f8-4acb-858c-a03f4f546299 response_code = 200 uri = /network-bootstore-config
38542024-07-25T05:54:25.486ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
38552024-07-25T05:54:25.514ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 59337 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:65253 req_id = fab19d2b-47b5-42e2-8009-8249d7f02a17 response_code = 204 uri = /switch-ports
38562024-07-25T05:54:25.515ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 142 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:65253 req_id = 72dacc4b-1d3f-4bc7-bb28-5aa9122f1b65 response_code = 200 uri = /network-bootstore-config
38572024-07-25T05:54:43.105ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 637 local_addr = [fd00:1122:3344:101::1]:57429 method = GET remote_addr = [fd00:1122:3344:101::d]:46986 req_id = c8bdbca9-4432-4e95-b1e9-e6ebb92892c2 response_code = 200 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376 uri = /8fa4bc78-1026-41f0-b09d-c67224071376
38582024-07-25T05:54:54.607ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:35945
38592024-07-25T05:54:54.607ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 782 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:35945 req_id = c94f3672-bdd0-4eba-9fdd-dd6e794221e0 response_code = 200 uri = /vpc-routes
38602024-07-25T05:54:54.614ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 129 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:35945 req_id = 8dbf994b-edd9-4d80-b289-95f7413440f7 response_code = 204 uri = /vpc-routes
38612024-07-25T05:54:54.620ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:48904
38622024-07-25T05:54:54.622ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 1831 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:48904 req_id = 092e4ba2-9b91-4386-9c9d-3a3ef54deb5b response_code = 200 uri = /v2p
38632024-07-25T05:54:54.814ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:50714
38642024-07-25T05:54:54.814ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
38652024-07-25T05:54:54.841ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
38662024-07-25T05:54:54.866ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 51626 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:50714 req_id = b2d6a866-3fe2-41b7-8668-e072ee48cc7d response_code = 204 uri = /switch-ports
38672024-07-25T05:54:54.866ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:36193
38682024-07-25T05:54:54.866ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 220 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:50714 req_id = 8d373ce2-6af8-4445-bb45-4e8ac360c6c1 response_code = 200 uri = /network-bootstore-config
38692024-07-25T05:54:54.867ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 981 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:36193 req_id = 2b5a6355-f8d9-40ea-8ec9-80801bc1acaf response_code = 200 uri = /vpc-routes
38702024-07-25T05:54:54.875ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 171 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:36193 req_id = f193489d-b36c-4593-8bc5-4af0a5b97563 response_code = 204 uri = /vpc-routes
38712024-07-25T05:54:54.896ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:62496
38722024-07-25T05:54:54.896ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 31 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:62496 req_id = ffd00c85-4f5a-4ef1-9ff2-3c7ee54da3b5 response_code = 200 uri = /vpc-routes
38732024-07-25T05:54:54.903ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:42442
38742024-07-25T05:54:54.904ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 1145 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:42442 req_id = 539b6cc1-ffae-4440-bafb-c90d52ff940f response_code = 200 uri = /v2p
38752024-07-25T05:54:54.904ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 58 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:62496 req_id = 2733d69a-6540-4c84-a6aa-68fe6b54ccc2 response_code = 204 uri = /vpc-routes
38762024-07-25T05:54:54.974ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:63997
38772024-07-25T05:54:54.974ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 860 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:63997 req_id = e3f8500d-7cac-4a04-8665-3e1ee35aeffd response_code = 200 uri = /v2p
38782024-07-25T05:54:55.078ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:46128
38792024-07-25T05:54:55.078ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
38802024-07-25T05:54:55.104ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
38812024-07-25T05:54:55.128ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 50512 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:46128 req_id = c2b5ecf5-09cb-45ce-90a2-a713d43ac6e4 response_code = 204 uri = /switch-ports
38822024-07-25T05:54:55.128ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:56699
38832024-07-25T05:54:55.129ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
38842024-07-25T05:54:55.129ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 887 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:46128 req_id = a8539df6-abc9-4b4c-bb47-ac34a2e84b19 response_code = 200 uri = /network-bootstore-config
38852024-07-25T05:54:55.157ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
38862024-07-25T05:54:55.183ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 54084 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:56699 req_id = 70759d14-35ee-4af8-a520-4a45c027ce59 response_code = 204 uri = /switch-ports
38872024-07-25T05:54:55.183ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 145 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:56699 req_id = 4f42d3a7-2f40-46b5-a5b3-a942fcc7c43e response_code = 200 uri = /network-bootstore-config
38882024-07-25T05:54:55.493ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:64593
38892024-07-25T05:54:55.494ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 523 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:64593 req_id = 18072aaa-0a5b-4f1c-aebb-142119778858 response_code = 200 uri = /baseboard
38902024-07-25T05:55:03.281ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:49630
38912024-07-25T05:55:03.282ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 652 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:49630 req_id = f238aaa5-fd50-4e4f-a7a8-dcfd791002e9 response_code = 200 uri = /v2p
38922024-07-25T05:55:03.282ZINFOSledAgent (PortManager): Mapping virtual NIC to physical host file = illumos-utils/src/opte/port_manager.rs:780 mapping = VirtualNetworkInterfaceHost { virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 243, 50, 113])), physical_host_ip: fd00:1122:3344:101::1, vni: Vni(9219180) }
38932024-07-25T05:55:03.282ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 120 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:49630 req_id = 920ba4da-91bc-4afd-8d2a-e6c5b7e02a39 response_code = 204 uri = /v2p
38942024-07-25T05:55:03.437ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62676
38952024-07-25T05:55:03.450ZINFOSledAgent (InstanceManager): ensuring instance is registered file = sled-agent/src/instance_manager.rs:577 hardware = InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: 21d50e0c-41c1-477c-ae5c-1216ff3e1c93, kind: Instance { id: 057c7948-d027-4c43-bcaf-62810b07fef3 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 243, 50, 113])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(9219180), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.1.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.1.176), floating_ips: [], firewall_rules: [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 21d50e0c-41c1-477c-ae5c-1216ff3e1c93, kind: Instance { id: 057c7948-d027-4c43-bcaf-62810b07fef3 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 243, 50, 113])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(9219180), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 21d50e0c-41c1-477c-ae5c-1216ff3e1c93, kind: Instance { id: 057c7948-d027-4c43-bcaf-62810b07fef3 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 243, 50, 113])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(9219180), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(9219180))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 21d50e0c-41c1-477c-ae5c-1216ff3e1c93, kind: Instance { id: 057c7948-d027-4c43-bcaf-62810b07fef3 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 243, 50, 113])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(9219180), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-e05b56d916a", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 4b57adae-59ff-4471-a606-c565e10339d7, read_only_parent: Some(Volume { block_size: 512, id: d266589e-cb07-4700-8896-7ca6fc087b4e, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 2a120dbc-0998-48d4-bfb5-e87686f7cf0c, key: Some("tAz9C6olO0l7NhRkBocuHL03NyM/pboGlS5gY9ANX2s="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::19]:19001", "[fd00:1122:3344:101::15]:19002", "[fd00:1122:3344:101::17]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 4b57adae-59ff-4471-a606-c565e10339d7, key: Some("EYyu4JaQfaGj+EquIBBl+na9yNkcVzIJU1YGUxTiCks="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::17]:19002", "[fd00:1122:3344:101::1a]:19001", "[fd00:1122:3344:101::13]:19000"] } }] } }], cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAACL4S75WPlYAADhLvlYAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAjOEu+Vj5WAAA4S75WAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiIwNTdjNzk0OC1kMDI3LTRjNDMtYmNhZi02MjgxMGIwN2ZlZjMiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJUDhENWJISjJLSUVxa2VTVm1YNUZ6Z21oMGVlbFU3MHpOY0p6bU1rbkgwNSJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA") } instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3 instance_runtime = InstanceRuntimeState { propolis_id: Some(a11cf978-990f-48d4-8597-cbd622e7cdbe (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-07-25T05:55:03.025890951Z } metadata = InstanceMetadata { silo_id: bb76b9e7-26a5-4e9e-a61c-3de62341dc71, project_id: 5b8f924c-3552-42d1-99ac-94cb0940fe4b } propolis_addr = [fd00:1122:3344:101::1:0]:12400 propolis_id = a11cf978-990f-48d4-8597-cbd622e7cdbe vmm_runtime = VmmRuntimeState { state: Starting, gen: Generation(1), time_updated: 2024-07-25T05:55:02.992660Z }
38962024-07-25T05:55:03.450ZINFOSledAgent (InstanceManager): registering new instance file = sled-agent/src/instance_manager.rs:611 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3 (instance)
38972024-07-25T05:55:03.452ZINFOSledAgent (InstanceManager): initializing new Instance file = sled-agent/src/instance.rs:951 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3 propolis_id = a11cf978-990f-48d4-8597-cbd622e7cdbe state = InstanceInitialState { hardware: InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: 21d50e0c-41c1-477c-ae5c-1216ff3e1c93, kind: Instance { id: 057c7948-d027-4c43-bcaf-62810b07fef3 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 243, 50, 113])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(9219180), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.1.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.1.176), floating_ips: [], firewall_rules: [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 21d50e0c-41c1-477c-ae5c-1216ff3e1c93, kind: Instance { id: 057c7948-d027-4c43-bcaf-62810b07fef3 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 243, 50, 113])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(9219180), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 21d50e0c-41c1-477c-ae5c-1216ff3e1c93, kind: Instance { id: 057c7948-d027-4c43-bcaf-62810b07fef3 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 243, 50, 113])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(9219180), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(9219180))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 21d50e0c-41c1-477c-ae5c-1216ff3e1c93, kind: Instance { id: 057c7948-d027-4c43-bcaf-62810b07fef3 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 243, 50, 113])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(9219180), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-e05b56d916a", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 4b57adae-59ff-4471-a606-c565e10339d7, read_only_parent: Some(Volume { block_size: 512, id: d266589e-cb07-4700-8896-7ca6fc087b4e, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 2a120dbc-0998-48d4-bfb5-e87686f7cf0c, key: Some("tAz9C6olO0l7NhRkBocuHL03NyM/pboGlS5gY9ANX2s="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::19]:19001", "[fd00:1122:3344:101::15]:19002", "[fd00:1122:3344:101::17]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 4b57adae-59ff-4471-a606-c565e10339d7, key: Some("EYyu4JaQfaGj+EquIBBl+na9yNkcVzIJU1YGUxTiCks="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::17]:19002", "[fd00:1122:3344:101::1a]:19001", "[fd00:1122:3344:101::13]:19000"] } }] } }], cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAACL4S75WPlYAADhLvlYAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAjOEu+Vj5WAAA4S75WAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiIwNTdjNzk0OC1kMDI3LTRjNDMtYmNhZi02MjgxMGIwN2ZlZjMiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJUDhENWJISjJLSUVxa2VTVm1YNUZ6Z21oMGVlbFU3MHpOY0p6bU1rbkgwNSJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA") }, instance_runtime: InstanceRuntimeState { propolis_id: Some(a11cf978-990f-48d4-8597-cbd622e7cdbe (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-07-25T05:55:03.025890951Z }, vmm_runtime: VmmRuntimeState { state: Starting, gen: Generation(1), time_updated: 2024-07-25T05:55:02.992660Z }, propolis_addr: [fd00:1122:3344:101::1:0]:12400 }
38982024-07-25T05:55:03.452ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 14873 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:62676 req_id = 7e64c280-b774-4d64-a0ca-9ee515e85f91 response_code = 200 uri = /instances/057c7948-d027-4c43-bcaf-62810b07fef3
38992024-07-25T05:55:03.631ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42469
39002024-07-25T05:55:03.632ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 1144 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:42469 req_id = ef9f21e4-6e18-4f58-8d4e-40ecb7e87c25 response_code = 200 uri = /instances/057c7948-d027-4c43-bcaf-62810b07fef3/state
39012024-07-25T05:55:03.633ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:426 port = Port { inner: PortInner(PortData { name: "opte6", ip: 172.30.0.5, mac: MacAddr6([168, 64, 37, 243, 50, 113]), slot: 0, vni: Vni { inner: 9219180 }, subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.0.1 } }) }
39022024-07-25T05:55:03.634ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
39032024-07-25T05:55:03.670ZINFOSledAgent (InstanceManager): Configuring new Omicron zone: oxz_propolis-server_a11cf978-990f-48d4-8597-cbd622e7cdbe file = illumos-utils/src/zone.rs:316 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3
39042024-07-25T05:55:03.694ZINFOSledAgent (InstanceManager): Installing Omicron zone: oxz_propolis-server_a11cf978-990f-48d4-8597-cbd622e7cdbe file = illumos-utils/src/zone.rs:354 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3
39052024-07-25T05:55:05.022ZINFOSledAgent (InstanceManager): Profile for oxz_propolis-server_a11cf978-990f-48d4-8597-cbd622e7cdbe: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="system/illumos/propolis-server"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlInstance0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1:0'/> <propval type="astring" name="listen_port" value='12400'/> <propval type="astring" name="metric_addr" value='[fd00:1122:3344:101::a]:12221'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3
39062024-07-25T05:55:05.022ZINFOSledAgent (InstanceManager): Booting oxz_propolis-server_a11cf978-990f-48d4-8597-cbd622e7cdbe zone file = illumos-utils/src/running_zone.rs:465 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3 zone = oxz_propolis-server_a11cf978-990f-48d4-8597-cbd622e7cdbe
39072024-07-25T05:55:05.220ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_a11cf978-990f-48d4-8597-cbd622e7cdbe") failed: Property not found. retry in 47.318565ms file = illumos-utils/src/svc.rs:36 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3 zone = oxz_propolis-server_a11cf978-990f-48d4-8597-cbd622e7cdbe
39082024-07-25T05:55:05.277ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_a11cf978-990f-48d4-8597-cbd622e7cdbe") failed: Property not found. retry in 85.301097ms file = illumos-utils/src/svc.rs:36 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3 zone = oxz_propolis-server_a11cf978-990f-48d4-8597-cbd622e7cdbe
39092024-07-25T05:55:05.378ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_a11cf978-990f-48d4-8597-cbd622e7cdbe") failed: Property not found. retry in 258.137467ms file = illumos-utils/src/svc.rs:36 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3 zone = oxz_propolis-server_a11cf978-990f-48d4-8597-cbd622e7cdbe
39102024-07-25T05:55:05.651ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_a11cf978-990f-48d4-8597-cbd622e7cdbe") failed: Property not found. retry in 290.194886ms file = illumos-utils/src/svc.rs:36 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3 zone = oxz_propolis-server_a11cf978-990f-48d4-8597-cbd622e7cdbe
39112024-07-25T05:55:05.952ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_a11cf978-990f-48d4-8597-cbd622e7cdbe") failed: Property not found. retry in 415.066175ms file = illumos-utils/src/svc.rs:36 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3 zone = oxz_propolis-server_a11cf978-990f-48d4-8597-cbd622e7cdbe
39122024-07-25T05:55:06.376ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_a11cf978-990f-48d4-8597-cbd622e7cdbe") failed: Property not found. retry in 576.853984ms file = illumos-utils/src/svc.rs:36 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3 zone = oxz_propolis-server_a11cf978-990f-48d4-8597-cbd622e7cdbe
39132024-07-25T05:55:06.965ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_a11cf978-990f-48d4-8597-cbd622e7cdbe") failed: Property not found. retry in 693.911549ms file = illumos-utils/src/svc.rs:36 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3 zone = oxz_propolis-server_a11cf978-990f-48d4-8597-cbd622e7cdbe
39142024-07-25T05:55:07.670ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_a11cf978-990f-48d4-8597-cbd622e7cdbe") failed: Property not found. retry in 779.486355ms file = illumos-utils/src/svc.rs:36 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3 zone = oxz_propolis-server_a11cf978-990f-48d4-8597-cbd622e7cdbe
39152024-07-25T05:55:08.460ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_a11cf978-990f-48d4-8597-cbd622e7cdbe") failed: Property not found. retry in 685.101643ms file = illumos-utils/src/svc.rs:36 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3 zone = oxz_propolis-server_a11cf978-990f-48d4-8597-cbd622e7cdbe
39162024-07-25T05:55:09.157ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_a11cf978-990f-48d4-8597-cbd622e7cdbe") failed: Property not found. retry in 1.171955963s file = illumos-utils/src/svc.rs:36 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3 zone = oxz_propolis-server_a11cf978-990f-48d4-8597-cbd622e7cdbe
39172024-07-25T05:55:10.358ZINFOSledAgent (InstanceManager): Started propolis in zone: oxz_propolis-server_a11cf978-990f-48d4-8597-cbd622e7cdbe file = sled-agent/src/instance.rs:1460 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3
39182024-07-25T05:55:10.367ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_a11cf978-990f-48d4-8597-cbd622e7cdbe") failed: Property not found. retry in 60.789127ms file = illumos-utils/src/svc.rs:36 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3
39192024-07-25T05:55:10.438ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_a11cf978-990f-48d4-8597-cbd622e7cdbe") failed: Property not found. retry in 60.826702ms file = illumos-utils/src/svc.rs:36 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3
39202024-07-25T05:55:10.509ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_a11cf978-990f-48d4-8597-cbd622e7cdbe") failed: Property not found. retry in 171.640499ms file = illumos-utils/src/svc.rs:36 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3
39212024-07-25T05:55:10.690ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_a11cf978-990f-48d4-8597-cbd622e7cdbe") failed: Property not found. retry in 226.970543ms file = illumos-utils/src/svc.rs:36 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3
39222024-07-25T05:55:10.929ZINFOSledAgent (InstanceManager): Propolis SMF service is online file = sled-agent/src/instance.rs:1469 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3
39232024-07-25T05:55:10.965ZINFOSledAgent (InstanceManager): Propolis HTTP server online file = sled-agent/src/instance.rs:1483 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3
39242024-07-25T05:55:10.968ZINFOSledAgent (InstanceManager): Sending ensure request to propolis: InstanceEnsureRequest { cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAACL4S75WPlYAADhLvlYAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAjOEu+Vj5WAAA4S75WAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiIwNTdjNzk0OC1kMDI3LTRjNDMtYmNhZi02MjgxMGIwN2ZlZjMiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJUDhENWJISjJLSUVxa2VTVm1YNUZ6Z21oMGVlbFU3MHpOY0p6bU1rbkgwNSJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"), disks: [DiskRequest { device: "nvme", name: "disk-e05b56d916a", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 4b57adae-59ff-4471-a606-c565e10339d7, read_only_parent: Some(Volume { block_size: 512, id: d266589e-cb07-4700-8896-7ca6fc087b4e, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 2a120dbc-0998-48d4-bfb5-e87686f7cf0c, key: Some("tAz9C6olO0l7NhRkBocuHL03NyM/pboGlS5gY9ANX2s="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::19]:19001", "[fd00:1122:3344:101::15]:19002", "[fd00:1122:3344:101::17]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 4b57adae-59ff-4471-a606-c565e10339d7, key: Some("EYyu4JaQfaGj+EquIBBl+na9yNkcVzIJU1YGUxTiCks="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::17]:19002", "[fd00:1122:3344:101::1a]:19001", "[fd00:1122:3344:101::13]:19000"] } }] } }], migrate: None, nics: [NetworkInterfaceRequest { name: "opte6", slot: Slot(0) }], properties: InstanceProperties { bootrom_id: 00000000-0000-0000-0000-000000000000, description: "Test description", id: 057c7948-d027-4c43-bcaf-62810b07fef3, image_id: 00000000-0000-0000-0000-000000000000, memory: 1024, metadata: InstanceMetadata { project_id: 5b8f924c-3552-42d1-99ac-94cb0940fe4b, silo_id: bb76b9e7-26a5-4e9e-a61c-3de62341dc71 }, name: "localshark", vcpus: 2 } } file = sled-agent/src/instance.rs:698 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3
39252024-07-25T05:55:11.188ZINFOSledAgent (InstanceManager): result of instance_ensure call is Ok(InstanceEnsureResponse { migrate: None }) file = sled-agent/src/instance.rs:700 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3
39262024-07-25T05:55:11.190ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Starting), migration_status: NoMigration, time: 2024-07-25T05:55:11.190034546Z } file = sled-agent/src/instance.rs:584 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3
39272024-07-25T05:55:11.190ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:606 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3 new_instance_state = InstanceRuntimeState { propolis_id: Some(a11cf978-990f-48d4-8597-cbd622e7cdbe (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-07-25T05:55:03.025890951Z } new_vmm_state = VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-07-25T05:55:11.190034546Z } propolis_id = a11cf978-990f-48d4-8597-cbd622e7cdbe
39282024-07-25T05:55:11.190ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:503 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(a11cf978-990f-48d4-8597-cbd622e7cdbe (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-07-25T05:55:03.025890951Z }, propolis_id: a11cf978-990f-48d4-8597-cbd622e7cdbe (propolis), vmm_state: VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-07-25T05:55:11.190034546Z }, migration_state: None }
39292024-07-25T05:55:11.286ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Running), migration_status: NoMigration, time: 2024-07-25T05:55:11.286801554Z } file = sled-agent/src/instance.rs:584 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3
39302024-07-25T05:55:11.286ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:606 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3 new_instance_state = InstanceRuntimeState { propolis_id: Some(a11cf978-990f-48d4-8597-cbd622e7cdbe (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-07-25T05:55:03.025890951Z } new_vmm_state = VmmRuntimeState { state: Running, gen: Generation(3), time_updated: 2024-07-25T05:55:11.286801554Z } propolis_id = a11cf978-990f-48d4-8597-cbd622e7cdbe
39312024-07-25T05:55:11.286ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:503 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(a11cf978-990f-48d4-8597-cbd622e7cdbe (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-07-25T05:55:03.025890951Z }, propolis_id: a11cf978-990f-48d4-8597-cbd622e7cdbe (propolis), vmm_state: VmmRuntimeState { state: Running, gen: Generation(3), time_updated: 2024-07-25T05:55:11.286801554Z }, migration_state: None }
39322024-07-25T05:55:11.324ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:56252
39332024-07-25T05:55:11.325ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 631 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:56252 req_id = e964c7fe-a81b-48b0-a5e7-7b3866a7be92 response_code = 200 uri = /vpc-routes
39342024-07-25T05:55:11.370ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 249 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:56252 req_id = 6ffa47ff-424a-4847-bf1c-62e02894beeb response_code = 204 uri = /vpc-routes
39352024-07-25T05:55:11.408ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63496
39362024-07-25T05:55:11.408ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 39 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:63496 req_id = 09080e83-f691-489d-8d2c-8e49b8cfd668 response_code = 200 uri = /vpc-routes
39372024-07-25T05:55:11.419ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 85 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:63496 req_id = 690227e3-c0a7-40cf-889a-5f339e6e728d response_code = 204 uri = /vpc-routes
39382024-07-25T05:55:13.238ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 596 local_addr = [fd00:1122:3344:101::1]:57429 method = GET remote_addr = [fd00:1122:3344:101::d]:46986 req_id = d7ec3b77-0923-4cb1-8212-32659a968203 response_code = 200 sled_id = 8fa4bc78-1026-41f0-b09d-c67224071376 uri = /8fa4bc78-1026-41f0-b09d-c67224071376
39392024-07-25T05:55:21.245ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59198
39402024-07-25T05:55:21.245ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 661 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:59198 req_id = 68a5c469-60a5-4133-a96e-ccabc9aa7524 response_code = 200 uri = /instances/057c7948-d027-4c43-bcaf-62810b07fef3/state
39412024-07-25T05:55:21.246ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Stopping), migration_status: NoMigration, time: 2024-07-25T05:55:21.246075796Z } file = sled-agent/src/instance.rs:584 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3
39422024-07-25T05:55:21.246ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:606 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3 new_instance_state = InstanceRuntimeState { propolis_id: Some(a11cf978-990f-48d4-8597-cbd622e7cdbe (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-07-25T05:55:03.025890951Z } new_vmm_state = VmmRuntimeState { state: Stopping, gen: Generation(5), time_updated: 2024-07-25T05:55:21.246075796Z } propolis_id = a11cf978-990f-48d4-8597-cbd622e7cdbe
39432024-07-25T05:55:21.246ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:503 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(a11cf978-990f-48d4-8597-cbd622e7cdbe (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-07-25T05:55:03.025890951Z }, propolis_id: a11cf978-990f-48d4-8597-cbd622e7cdbe (propolis), vmm_state: VmmRuntimeState { state: Stopping, gen: Generation(5), time_updated: 2024-07-25T05:55:21.246075796Z }, migration_state: None }
39442024-07-25T05:55:21.337ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Destroyed), migration_status: NoMigration, time: 2024-07-25T05:55:21.337558358Z } file = sled-agent/src/instance.rs:584 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3
39452024-07-25T05:55:21.337ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:606 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3 new_instance_state = InstanceRuntimeState { propolis_id: None, dst_propolis_id: None, migration_id: None, gen: Generation(5), time_updated: 2024-07-25T05:55:21.337558358Z } new_vmm_state = VmmRuntimeState { state: Destroyed, gen: Generation(6), time_updated: 2024-07-25T05:55:21.337558358Z } propolis_id = a11cf978-990f-48d4-8597-cbd622e7cdbe
39462024-07-25T05:55:21.337ZINFOSledAgent (InstanceManager): terminating VMM that has exited file = sled-agent/src/instance.rs:621 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3
39472024-07-25T05:55:21.337ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
39482024-07-25T05:55:21.337ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
39492024-07-25T05:55:21.337ZINFOSledAgent (ZoneBundler): creating zone bundle context = ZoneBundleContext { storage_dirs: ["/pool/int/6d2aa15e-618e-402e-b368-cf27fd8def68/debug/bundle/zone", "/pool/int/75c788e1-45ef-4fcf-958d-db07331bef52/debug/bundle/zone"], cause: TerminatedInstance, extra_log_dirs: ["/pool/ext/dd1c81eb-5ddf-42bb-acfe-348d2d141fca/crypt/debug", "/pool/ext/59d3cc5a-40b7-44b8-b2af-cfc21755b8bf/crypt/debug", "/pool/ext/e78b4f40-df20-4bcf-bd8a-b80afa109d70/crypt/debug", "/pool/ext/63a12260-6721-4f51-85d5-262aedc90628/crypt/debug", "/pool/ext/587239d7-a145-4bb7-b24a-aff014195deb/crypt/debug", "/pool/ext/50e63493-6b9c-42de-846b-797a312b5fbc/crypt/debug", "/pool/ext/fd75587e-5d72-48df-a44e-8602b2d61469/crypt/debug", "/pool/ext/20d995c8-4908-4380-8438-c04a6e3adea2/crypt/debug", "/pool/ext/e43689a1-378e-4be8-9661-3bc522647c6d/crypt/debug"] } file = sled-agent/src/zone_bundle.rs:445 zone_name = oxz_propolis-server_a11cf978-990f-48d4-8597-cbd622e7cdbe
39502024-07-25T05:55:21.375ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:64935
39512024-07-25T05:55:21.376ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 887 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:64935 req_id = 64b7b6bb-8636-40dc-9986-3302fcc88b23 response_code = 200 uri = /vpc-routes
39522024-07-25T05:55:21.389ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 181 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:64935 req_id = 0e7245a1-f108-41d2-b1eb-e8782e80738d response_code = 204 uri = /vpc-routes
39532024-07-25T05:55:22.129ZINFOSledAgent (ZoneBundler): finished zone bundle file = sled-agent/src/zone_bundle.rs:1246 metadata = ZoneBundleMetadata { id: ZoneBundleId { zone_name: "oxz_propolis-server_a11cf978-990f-48d4-8597-cbd622e7cdbe", bundle_id: 295d4e51-3915-42ac-973a-c4827e4b067e }, time_created: 2024-07-25T05:55:21.338268985Z, version: 0, cause: TerminatedInstance }
39542024-07-25T05:55:22.129ZWARNSledAgent (InstanceManager): Halting and removing zone: oxz_propolis-server_a11cf978-990f-48d4-8597-cbd622e7cdbe file = sled-agent/src/instance.rs:793 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3
39552024-07-25T05:55:23.912ZINFOSledAgent (InstanceManager): halt_and_remove_logged: Previous zone state: Running file = illumos-utils/src/zone.rs:270 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3
39562024-07-25T05:55:23.928ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:906 id = RouterId { vni: Vni(9219180), kind: System }
39572024-07-25T05:55:23.928ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:906 id = RouterId { vni: Vni(9219180), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }
39582024-07-25T05:55:23.929ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:503 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: None, dst_propolis_id: None, migration_id: None, gen: Generation(5), time_updated: 2024-07-25T05:55:21.337558358Z }, propolis_id: a11cf978-990f-48d4-8597-cbd622e7cdbe (propolis), vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-07-25T05:55:23.929142718Z }, migration_state: None }
39592024-07-25T05:55:24.010ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:41256
39602024-07-25T05:55:24.010ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 219 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:41256 req_id = 38051a53-22d9-4872-a0c4-eefc4c28fce8 response_code = 200 uri = /v2p
39612024-07-25T05:55:24.011ZINFOSledAgent (PortManager): Mapping virtual NIC to physical host file = illumos-utils/src/opte/port_manager.rs:780 mapping = VirtualNetworkInterfaceHost { virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 243, 50, 113])), physical_host_ip: fd00:1122:3344:101::1, vni: Vni(9219180) }
39622024-07-25T05:55:24.011ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 129 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:41256 req_id = 5b62786b-f36b-46c5-a35f-9ba45e263c30 response_code = 204 uri = /v2p
39632024-07-25T05:55:24.512ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:55738
39642024-07-25T05:55:24.520ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Not Found error_message_internal = No such instance ID: 057c7948-d027-4c43-bcaf-62810b07fef3 file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 911 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:55738 req_id = c5d8cf51-cca1-4f39-bfec-c8f88eecbb87 response_code = 404 uri = /instances/057c7948-d027-4c43-bcaf-62810b07fef3/state
39652024-07-25T05:55:24.740ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:64972
39662024-07-25T05:55:24.740ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 73 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:64972 req_id = e894df83-4e8e-4b94-9347-7432bbdc8fba response_code = 200 uri = /vpc-routes
39672024-07-25T05:55:24.746ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:503 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: None, dst_propolis_id: None, migration_id: None, gen: Generation(5), time_updated: 2024-07-25T05:55:21.337558358Z }, propolis_id: a11cf978-990f-48d4-8597-cbd622e7cdbe (propolis), vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-07-25T05:55:23.929142718Z }, migration_state: None }
39682024-07-25T05:55:24.747ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 201 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:64972 req_id = dbc39164-9c2e-49df-83d8-8c884ab2d3f5 response_code = 204 uri = /vpc-routes
39692024-07-25T05:55:24.750ZINFOSledAgent (InstanceManager): State monitoring task complete file = sled-agent/src/instance.rs:736 instance_id = 057c7948-d027-4c43-bcaf-62810b07fef3
39702024-07-25T05:55:24.774ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:39154
39712024-07-25T05:55:24.774ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 138 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:39154 req_id = c7daba7f-c4fc-4149-8c46-2b003ddd5a1a response_code = 200 uri = /v2p
39722024-07-25T05:55:24.964ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:60946
39732024-07-25T05:55:24.964ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 60 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:60946 req_id = 0361620a-9789-49d0-a517-5b226feaf1ec response_code = 200 uri = /vpc-routes
39742024-07-25T05:55:24.971ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 94 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:60946 req_id = 713115ff-39ac-4cb8-a836-143958c84fbc response_code = 204 uri = /vpc-routes
39752024-07-25T05:55:25.021ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:53414
39762024-07-25T05:55:25.021ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:53414 req_id = 8e9c8cdc-d61f-414e-8abb-ffa8f98e5323 response_code = 200 uri = /v2p
39772024-07-25T05:55:25.027ZINFOSledAgent (PortManager): Clearing mapping of virtual NIC to physical host file = illumos-utils/src/opte/port_manager.rs:820 mapping = VirtualNetworkInterfaceHost { virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 243, 50, 113])), physical_host_ip: fd00:1122:3344:101::1, vni: Vni(9219180) }
39782024-07-25T05:55:25.027ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 6147 local_addr = [fd00:1122:3344:101::1]:12345 method = DELETE remote_addr = [fd00:1122:3344:101::b]:53414 req_id = dbc6d02a-baee-4a2a-92b0-554786c2f403 response_code = 204 uri = /v2p
39792024-07-25T05:55:25.063ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:49626
39802024-07-25T05:55:25.063ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
39812024-07-25T05:55:25.093ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
39822024-07-25T05:55:25.118ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 55372 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:49626 req_id = 403e50ca-1d1b-4e93-8615-6f9e5a8aca60 response_code = 204 uri = /switch-ports
39832024-07-25T05:55:25.119ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:58369
39842024-07-25T05:55:25.119ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:58381
39852024-07-25T05:55:25.119ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:58369 req_id = 1e4fed08-c16c-4a19-b020-ad2e669545b6 response_code = 200 uri = /vpc-routes
39862024-07-25T05:55:25.119ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 223 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:49626 req_id = 8962ddb6-5be9-4c5a-9c0d-5bea2f1a9185 response_code = 200 uri = /network-bootstore-config
39872024-07-25T05:55:25.119ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 245 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:58381 req_id = 40a41601-ec37-43c9-a802-e9ef0b84ca5a response_code = 200 uri = /v2p
39882024-07-25T05:55:25.127ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 825 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:58369 req_id = f38e0d2e-13b4-4d43-abb7-4a8829828050 response_code = 204 uri = /vpc-routes
39892024-07-25T05:55:25.248ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37824
39902024-07-25T05:55:25.248ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
39912024-07-25T05:55:25.275ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
39922024-07-25T05:55:25.299ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 50655 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:37824 req_id = cd17efcb-1c77-4efb-a03f-7a3c87291fc3 response_code = 204 uri = /switch-ports
39932024-07-25T05:55:25.299ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:59740
39942024-07-25T05:55:25.299ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 58 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:59740 req_id = 4563e512-52a7-40c6-ae4f-a807bf0ed37c response_code = 200 uri = /vpc-routes
39952024-07-25T05:55:25.299ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 84 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:37824 req_id = 0c292f91-818c-49ee-bfc0-6ecc9bbd020f response_code = 200 uri = /network-bootstore-config
39962024-07-25T05:55:25.306ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 127 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:59740 req_id = 9e1288d6-fb6c-41cf-9afb-272ec581ba0e response_code = 204 uri = /vpc-routes
39972024-07-25T05:55:25.325ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:44630
39982024-07-25T05:55:25.325ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3824
39992024-07-25T05:55:25.349ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3835
40002024-07-25T05:55:25.372ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 47000 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:44630 req_id = 594ce8f2-d127-4bb8-b228-54ee3cf34733 response_code = 204 uri = /switch-ports
40012024-07-25T05:55:25.372ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 86 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:44630 req_id = e5276950-241b-42cd-8e3f-6d27dcfbf82c response_code = 200 uri = /network-bootstore-config
40022024-07-25T05:55:25.533ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:63648
40032024-07-25T05:55:25.533ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 556 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:63648 req_id = e0abcac7-d022-4424-ba44-1b3f2a33c1ca response_code = 200 uri = /baseboard
40042024-07-25T05:55:25.723ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33615
40052024-07-25T05:55:25.723ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 170 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:33615 req_id = d71bdeb8-e01a-411a-b1c5-470d76ad7328 response_code = 200 uri = /v2p
40062024-07-25T05:55:27.666ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62550
40072024-07-25T05:55:27.667ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 638 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:62550 req_id = 72ba84e9-1530-4813-8891-25bfa1efbefc response_code = 200 uri = /vpc-routes
40082024-07-25T05:55:27.673ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 122 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:62550 req_id = 2aefdf84-30fd-433c-a798-94507af00945 response_code = 204 uri = /vpc-routes