Max level shown:
1 [ Jul 19 17:44:16 Enabled. ]
2 [ Jul 19 17:44:16 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]
3 [ Jul 19 17:44:16 Method "start" exited with status 0. ]
4 [ Jul 19 17:44:16 Rereading configuration. ]
5 [ Jul 19 17:44:16 No 'refresh' method defined. Treating as :true. ]
6 note: configured to log to "/dev/stdout"
72024-07-19T17:44:16.305ZINFOSledAgent: Ensuring contract reaper thread file = illumos-utils/src/running_zone.rs:110
82024-07-19T17:44:16.306ZINFOSledAgent (BootstrapAgentStartup): finding links ["net0", "net1"] file = sled-agent/src/bootstrap/pre_server.rs:255
92024-07-19T17:44:16.359ZINFOSledAgent (BootstrapAgentStartup): Starting mg-ddm service addr-objs = [AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }] file = sled-agent/src/bootstrap/pre_server.rs:262
102024-07-19T17:44:16.359ZINFOSledAgent (BootstrapAgentStartup): Importing mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:42 path = /opt/oxide/mg-ddm/pkg/ddm/manifest.xml
112024-07-19T17:44:16.473ZINFOSledAgent (BootstrapAgentStartup): Setting mg-ddm interfaces file = sled-agent/src/bootstrap/maghemite.rs:50 interfaces = ("net0/ll" "net1/ll")
122024-07-19T17:44:16.483ZINFOSledAgent (BootstrapAgentStartup): Enabling mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:56
132024-07-19T17:44:16.491ZINFOSledAgent (BootstrapAgentStartup): Ensuring zfs key directory exists file = sled-agent/src/bootstrap/pre_server.rs:272 path = /var/run/oxide/
142024-07-19T17:44:17.091ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fdb0:18c0:4d0d:9fb2::, width: 64 }
152024-07-19T17:44:17.231ZINFOSledAgent: Starting KeyManager file = sled-agent/src/long_running_tasks.rs:125
162024-07-19T17:44:17.231ZINFOSledAgent: Starting StorageManager file = sled-agent/src/long_running_tasks.rs:137
172024-07-19T17:44:17.231ZINFOSledAgent: Starting StorageMonitor file = sled-agent/src/long_running_tasks.rs:150
182024-07-19T17:44:17.231ZINFOSledAgent: Starting HardwareManager file = sled-agent/src/long_running_tasks.rs:172 nongimlet_observed_disks = [] sled_mode = Scrimlet { asic: SoftNpuZone }
192024-07-19T17:44:17.231ZINFOSledAgent (DumpSetup-worker): DumpSetup poll loop started. file = sled-agent/src/dump_setup.rs:520
202024-07-19T17:44:17.231ZINFOSledAgent (HardwareManager): Creating HardwareManager file = sled-hardware/src/illumos/mod.rs:718
212024-07-19T17:44:17.234ZINFOSledAgent (HardwareManager): Generated i86pc baseboard Pc { identifier: "buskin", model: "i86pc" } file = sled-hardware/src/illumos/mod.rs:625
222024-07-19T17:44:17.234ZWARNSledAgent (HardwareManager): Device is not a Gimlet (i86pc), proceeding with null hardware view file = sled-hardware/src/illumos/mod.rs:767
232024-07-19T17:44:17.234ZINFOSledAgent: Starting HardwareMonitor file = sled-agent/src/long_running_tasks.rs:186
242024-07-19T17:44:17.234ZINFOSledAgent (HardwareManager): Monitoring for hardware updates file = sled-hardware/src/illumos/mod.rs:826
252024-07-19T17:44:17.234ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/m2_0.vdev
262024-07-19T17:44:17.234ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = None
272024-07-19T17:44:17.234ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:404
282024-07-19T17:44:17.235ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
292024-07-19T17:44:17.235ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:127 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } path = /scratch/m2_0.vdev
302024-07-19T17:44:17.246ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_0.vdev file = sled-hardware/src/disk.rs:361
312024-07-19T17:44:17.246ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:372 id = 301424fa-c483-4ce9-92fb-3c8ee10e6f9f (zpool)
322024-07-19T17:44:17.344ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 301424fa-c483-4ce9-92fb-3c8ee10e6f9f (zpool), kind: Internal }
332024-07-19T17:44:17.554ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:404
342024-07-19T17:44:17.554ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/m2_1.vdev
352024-07-19T17:44:17.554ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
362024-07-19T17:44:17.554ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
372024-07-19T17:44:17.554ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:404
382024-07-19T17:44:17.554ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
392024-07-19T17:44:17.554ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:127 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } path = /scratch/m2_1.vdev
402024-07-19T17:44:17.563ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_1.vdev file = sled-hardware/src/disk.rs:361
412024-07-19T17:44:17.563ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:372 id = 20dccfdd-e68a-468d-8629-7f7e0834aa1c (zpool)
422024-07-19T17:44:17.653ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 20dccfdd-e68a-468d-8629-7f7e0834aa1c (zpool), kind: Internal }
432024-07-19T17:44:17.862ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_0.vdev
442024-07-19T17:44:17.862ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 301424fa-c483-4ce9-92fb-3c8ee10e6f9f (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 20dccfdd-e68a-468d-8629-7f7e0834aa1c (zpool), kind: Internal } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
452024-07-19T17:44:17.862ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
462024-07-19T17:44:17.862ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
472024-07-19T17:44:17.862ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:404
482024-07-19T17:44:17.862ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
492024-07-19T17:44:17.862ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_1.vdev
502024-07-19T17:44:17.862ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:404
512024-07-19T17:44:17.862ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 301424fa-c483-4ce9-92fb-3c8ee10e6f9f (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 20dccfdd-e68a-468d-8629-7f7e0834aa1c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
522024-07-19T17:44:17.862ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
532024-07-19T17:44:17.862ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
542024-07-19T17:44:17.862ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
552024-07-19T17:44:17.862ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_2.vdev
562024-07-19T17:44:17.862ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:404
572024-07-19T17:44:17.862ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
582024-07-19T17:44:17.862ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 301424fa-c483-4ce9-92fb-3c8ee10e6f9f (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 20dccfdd-e68a-468d-8629-7f7e0834aa1c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
592024-07-19T17:44:17.862ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_3.vdev
602024-07-19T17:44:17.862ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
612024-07-19T17:44:17.862ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
622024-07-19T17:44:17.862ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:404
632024-07-19T17:44:17.862ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
642024-07-19T17:44:17.862ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 301424fa-c483-4ce9-92fb-3c8ee10e6f9f (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 20dccfdd-e68a-468d-8629-7f7e0834aa1c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
652024-07-19T17:44:17.862ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
662024-07-19T17:44:17.862ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
672024-07-19T17:44:17.862ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_4.vdev
682024-07-19T17:44:17.862ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 301424fa-c483-4ce9-92fb-3c8ee10e6f9f (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 20dccfdd-e68a-468d-8629-7f7e0834aa1c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
692024-07-19T17:44:17.862ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:404
702024-07-19T17:44:17.862ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
712024-07-19T17:44:17.862ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
722024-07-19T17:44:17.862ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
732024-07-19T17:44:17.862ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_5.vdev
742024-07-19T17:44:17.862ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:404
752024-07-19T17:44:17.862ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 301424fa-c483-4ce9-92fb-3c8ee10e6f9f (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 20dccfdd-e68a-468d-8629-7f7e0834aa1c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
762024-07-19T17:44:17.862ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
772024-07-19T17:44:17.862ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
782024-07-19T17:44:17.862ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
792024-07-19T17:44:17.862ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_6.vdev
802024-07-19T17:44:17.862ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 301424fa-c483-4ce9-92fb-3c8ee10e6f9f (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 20dccfdd-e68a-468d-8629-7f7e0834aa1c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
812024-07-19T17:44:17.862ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
822024-07-19T17:44:17.862ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:404
832024-07-19T17:44:17.862ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
842024-07-19T17:44:17.862ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
852024-07-19T17:44:17.862ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_7.vdev
862024-07-19T17:44:17.862ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:404
872024-07-19T17:44:17.862ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
882024-07-19T17:44:17.862ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 301424fa-c483-4ce9-92fb-3c8ee10e6f9f (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 20dccfdd-e68a-468d-8629-7f7e0834aa1c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
892024-07-19T17:44:17.862ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
902024-07-19T17:44:17.862ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
912024-07-19T17:44:17.862ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_8.vdev
922024-07-19T17:44:17.862ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:404
932024-07-19T17:44:17.862ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
942024-07-19T17:44:17.862ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 301424fa-c483-4ce9-92fb-3c8ee10e6f9f (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 20dccfdd-e68a-468d-8629-7f7e0834aa1c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
952024-07-19T17:44:17.862ZINFOSledAgent: Waiting for boot disk file = sled-agent/src/long_running_tasks.rs:96
962024-07-19T17:44:17.862ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
972024-07-19T17:44:17.862ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
982024-07-19T17:44:17.862ZINFOSledAgent: Found boot disk DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-agent/src/long_running_tasks.rs:98
992024-07-19T17:44:17.862ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
1002024-07-19T17:44:17.862ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 301424fa-c483-4ce9-92fb-3c8ee10e6f9f (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 20dccfdd-e68a-468d-8629-7f7e0834aa1c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
1012024-07-19T17:44:17.862ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
1022024-07-19T17:44:17.862ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
1032024-07-19T17:44:17.863ZINFOSledAgent: Starting Bootstore file = sled-agent/src/long_running_tasks.rs:210
1042024-07-19T17:44:17.863ZINFOSledAgent (bootstore): No ledger in /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1052024-07-19T17:44:17.863ZINFOSledAgent (bootstore): No ledger in /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1062024-07-19T17:44:17.863ZINFOSledAgent (bootstore): No ledger found. Loading Fsm::State as Uninitialized file = bootstore/src/schemes/v0/storage.rs:83 peer_id = pc-buskin-i86pc
1072024-07-19T17:44:17.863ZINFOSledAgent (bootstore): No ledger in /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1082024-07-19T17:44:17.863ZINFOSledAgent (bootstore): No ledger in /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1092024-07-19T17:44:17.863ZINFOSledAgent (bootstore): No ledger found for network config file = bootstore/src/schemes/v0/storage.rs:146 peer_id = pc-buskin-i86pc
1102024-07-19T17:44:17.863ZINFOSledAgent: Starting Bootstore DDMD poller file = sled-agent/src/long_running_tasks.rs:215
1112024-07-19T17:44:17.863ZINFOSledAgent: Starting ZoneBundler related tasks file = sled-agent/src/long_running_tasks.rs:230
1122024-07-19T17:44:17.873ZINFOSledAgent (ServiceManager): Creating ServiceManager file = sled-agent/src/services.rs:722
1132024-07-19T17:44:17.874ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:292 next_cleanup = Instant { tv_sec: 2360, tv_nsec: 168828361 } time_to_next_cleanup = 599.999905848s
1142024-07-19T17:44:17.874ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3583
1152024-07-19T17:44:17.874ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
1162024-07-19T17:44:17.874ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/sled-agent-request.json file = common/src/ledger.rs:188
1172024-07-19T17:44:17.874ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/sled-agent-request.json file = common/src/ledger.rs:188
1182024-07-19T17:44:17.877ZINFOSledAgent (dropshot (BootstrapAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:205 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80
1192024-07-19T17:44:17.877ZINFOSledAgent (SledAgentSprocketsServer): Started listening file = sled-agent/src/bootstrap/sprockets_server.rs:44 local_addr = [fdb0:18c0:4d0d:9fb2::1]:12346
1202024-07-19T17:44:17.877ZINFOSledAgent (RSS): Injecting RSS configuration: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.1.164, last: 10.151.1.173, }, ), ], external_dns_ips: [ 10.151.1.164, 10.151.1.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.1.100, vlan_id: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.1.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, } file = sled-agent/src/rack_setup/service.rs:1011
1212024-07-19T17:44:17.877ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
1222024-07-19T17:44:17.877ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
1232024-07-19T17:44:17.877ZINFOSledAgent (RSS): No ledger in /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1242024-07-19T17:44:17.877ZINFOSledAgent (RSS): No ledger in /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1252024-07-19T17:44:17.877ZINFOSledAgent (RSS): RSS configuration has not been fully applied yet file = sled-agent/src/rack_setup/service.rs:1071
1262024-07-19T17:44:17.877ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
1272024-07-19T17:44:17.877ZINFOSledAgent (RSS): No ledger in /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/rss-sled-plan.json file = common/src/ledger.rs:188
1282024-07-19T17:44:17.877ZINFOSledAgent (RSS): No ledger in /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/rss-sled-plan.json file = common/src/ledger.rs:188
1292024-07-19T17:44:17.877ZINFOSledAgent (RSS): Creating new allocation plan file = sled-agent/src/rack_setup/service.rs:1112
1302024-07-19T17:44:17.877ZINFOSledAgent (RSS): Serializing plan file = sled-agent/src/rack_setup/plan/sled.rs:156
1312024-07-19T17:44:17.877ZINFOSledAgent (RSS): Creating plan for the sled at fdb0:18c0:4d0d:9fb2::1 file = sled-agent/src/rack_setup/plan/sled.rs:129
1322024-07-19T17:44:17.877ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
1332024-07-19T17:44:17.877ZINFOSledAgent (RSS): Writing ledger to /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1342024-07-19T17:44:17.878ZINFOSledAgent (RSS): Writing ledger to /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1352024-07-19T17:44:17.878ZINFOSledAgent (RSS): Sled plan written to storage file = sled-agent/src/rack_setup/plan/sled.rs:176
1362024-07-19T17:44:17.878ZINFOSledAgent (RSS): Writing Rack Network Configuration to bootstore file = sled-agent/src/rack_setup/service.rs:1145
1372024-07-19T17:44:17.878ZINFOSledAgent (bootstore): Attempting to update network config with generation: 1, current_generation: 0 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-buskin-i86pc
1382024-07-19T17:44:17.878ZINFOSledAgent (bootstore): Writing ledger to /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
1392024-07-19T17:44:17.878ZINFOSledAgent (bootstore): Writing ledger to /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
1402024-07-19T17:44:17.878ZINFOSledAgent (bootstore): Broadcasting network config with generation 1 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-buskin-i86pc
1412024-07-19T17:44:17.879ZINFOSledAgent (BootstrapAgentRssHandler): Received initialization request from RSS file = sled-agent/src/bootstrap/rss_handle.rs:217 request = StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 0c1108fb-cc91-4535-adf7-846d50ef8716, rack_id: 0ac29522-8a9f-4432-bdfe-a8adde17b3b5, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } target_sled = [fdb0:18c0:4d0d:9fb2::1]:12346
1422024-07-19T17:44:17.879ZINFOSledAgent (SledAgentSprocketsServer): Accepted connection file = sled-agent/src/bootstrap/sprockets_server.rs:66 remote_addr = [fdb0:18c0:4d0d:9fb2::1]:60684
1432024-07-19T17:44:17.879ZINFOSledAgent (SledAgentMain): Loading Sled Agent: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 0c1108fb-cc91-4535-adf7-846d50ef8716, rack_id: 0ac29522-8a9f-4432-bdfe-a8adde17b3b5, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } file = sled-agent/src/bootstrap/server.rs:360
1442024-07-19T17:44:17.879ZINFOSledAgent (SledAgentMain): KeyManager: using hardcoded secret retriever file = sled-agent/src/bootstrap/server.rs:375
1452024-07-19T17:44:17.879ZINFOSledAgent: setting up sled agent server file = sled-agent/src/server.rs:44
1462024-07-19T17:44:17.879ZINFOSledAgent (StorageManager): Received KeyManagerReady file = sled-storage/src/manager.rs:404
1472024-07-19T17:44:17.879ZINFOSledAgent (StorageManager): Loading storage config file = sled-storage/src/manager.rs:572
1482024-07-19T17:44:17.879ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
1492024-07-19T17:44:17.879ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:101::, width: 64 }
1502024-07-19T17:44:17.879ZINFOSledAgent (StorageManager): No ledger in /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_ledger
1512024-07-19T17:44:17.879ZINFOSledAgent (StorageManager): No ledger in /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_ledger
1522024-07-19T17:44:17.879ZINFOSledAgent (StorageManager): No ledger of physical disks exists file = sled-storage/src/manager.rs:561
1532024-07-19T17:44:17.879ZINFOSledAgent (StorageManager): KeyManager ready, but no ledger detected file = sled-storage/src/manager.rs:589
1542024-07-19T17:44:17.879ZINFOSledAgent (StorageManager): automatically managing disks file = sled-storage/src/manager.rs:393
1552024-07-19T17:44:17.879ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:374
1562024-07-19T17:44:17.879ZINFOSledAgent (StorageManager): Successfully synchronized disks without error file = sled-storage/src/manager.rs:479 result = DisksManagementResult { status: [] }
1572024-07-19T17:44:17.881ZINFOSledAgent (ServiceManager): Enabling switch zone (new) file = sled-agent/src/services.rs:3815
1582024-07-19T17:44:17.913ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
1592024-07-19T17:44:17.934ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:316
1602024-07-19T17:44:17.952ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:354
1612024-07-19T17:44:17.997ZINFOSledAgent: SledAgent::new(..) starting file = sled-agent/src/sled_agent.rs:366 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
1622024-07-19T17:44:17.997ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
1632024-07-19T17:44:17.997ZINFOSledAgent: Requested swap device of size 64 GiB file = sled-agent/src/sled_agent.rs:378 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
1642024-07-19T17:44:17.997ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1652024-07-19T17:44:18.000ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 50.840126ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "68a98f37-3ce8-479e-82e7-f6f6c6391517", "content-length": "124", "date": "Fri, 19 Jul 2024 17:44:17 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "68a98f37-3ce8-479e-82e7-f6f6c6391517" } file = clients/ddm-admin-client/src/lib.rs:177
1662024-07-19T17:44:18.010ZINFOSledAgent: attempting to create encrypted zvol: name="oxi_301424fa-c483-4ce9-92fb-3c8ee10e6f9f/swap", size_gb=64 file = sled-agent/src/swap_device.rs:153
1672024-07-19T17:44:18.035ZINFOSledAgent: successfully created encrypted zvol: name="oxi_301424fa-c483-4ce9-92fb-3c8ee10e6f9f/swap", size_gb=64 file = sled-agent/src/swap_device.rs:249
1682024-07-19T17:44:18.035ZINFOSledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_301424fa-c483-4ce9-92fb-3c8ee10e6f9f/swap" file = sled-agent/src/swap_device.rs:96
1692024-07-19T17:44:18.040ZINFOSledAgent: Mounting backing filesystems file = sled-agent/src/sled_agent.rs:393 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
1702024-07-19T17:44:18.041ZINFOSledAgent (BackingFs): Processing fmd file = sled-agent/src/backing_fs.rs:126
1712024-07-19T17:44:18.041ZINFOSledAgent (BackingFs): Ensuring dataset oxi_301424fa-c483-4ce9-92fb-3c8ee10e6f9f/backing/fmd file = sled-agent/src/backing_fs.rs:136
1722024-07-19T17:44:18.053ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1732024-07-19T17:44:18.053ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 72.232124ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "a652dc9c-2a64-4f99-b5c3-7a5b6f0aa923", "content-length": "124", "date": "Fri, 19 Jul 2024 17:44:17 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "a652dc9c-2a64-4f99-b5c3-7a5b6f0aa923" } file = clients/ddm-admin-client/src/lib.rs:177
1742024-07-19T17:44:18.115ZINFOSledAgent (BackingFs): Stopping service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:170
1752024-07-19T17:44:18.127ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1762024-07-19T17:44:18.128ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 137.365348ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "7db990ec-e77b-48bc-a8d9-f3174b000959", "content-length": "124", "date": "Fri, 19 Jul 2024 17:44:18 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "7db990ec-e77b-48bc-a8d9-f3174b000959" } file = clients/ddm-admin-client/src/lib.rs:177
1772024-07-19T17:44:18.266ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1782024-07-19T17:44:18.267ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 120.102669ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "35c0ef17-3dc8-45b9-8a3c-0b74aee0603b", "content-length": "124", "date": "Fri, 19 Jul 2024 17:44:18 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "35c0ef17-3dc8-45b9-8a3c-0b74aee0603b" } file = clients/ddm-admin-client/src/lib.rs:177
1792024-07-19T17:44:18.334ZINFOSledAgent (BackingFs): Mounting oxi_301424fa-c483-4ce9-92fb-3c8ee10e6f9f/backing/fmd on /var/fm/fmd file = sled-agent/src/backing_fs.rs:178
1802024-07-19T17:44:18.347ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/rsrc file = sled-agent/src/backing_fs.rs:186
1812024-07-19T17:44:18.347ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/ckpt file = sled-agent/src/backing_fs.rs:186
1822024-07-19T17:44:18.347ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/xprt file = sled-agent/src/backing_fs.rs:186
1832024-07-19T17:44:18.347ZINFOSledAgent (BackingFs): Starting service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:192
1842024-07-19T17:44:18.388ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1852024-07-19T17:44:18.388ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 104.992291ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "535f9f29-935e-44c3-b529-ff7eada01c20", "content-length": "124", "date": "Fri, 19 Jul 2024 17:44:18 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "535f9f29-935e-44c3-b529-ff7eada01c20" } file = clients/ddm-admin-client/src/lib.rs:177
1862024-07-19T17:44:18.494ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1872024-07-19T17:44:18.495ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 157.48605ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "08626959-1068-4cfb-8914-aec32e8af99d", "content-length": "124", "date": "Fri, 19 Jul 2024 17:44:18 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "08626959-1068-4cfb-8914-aec32e8af99d" } file = clients/ddm-admin-client/src/lib.rs:177
1882024-07-19T17:44:18.653ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1892024-07-19T17:44:18.654ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 378.194517ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "f9741c56-15e2-494f-a276-b3cb428dab8a", "content-length": "124", "date": "Fri, 19 Jul 2024 17:44:18 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "f9741c56-15e2-494f-a276-b3cb428dab8a" } file = clients/ddm-admin-client/src/lib.rs:177
1902024-07-19T17:44:19.033ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1912024-07-19T17:44:19.073ZINFOSledAgent: using '[AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }]' as data links for xde driver file = illumos-utils/src/opte/illumos.rs:90 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
1922024-07-19T17:44:19.073ZINFOSledAgent (dropshot): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:205 local_addr = [fd00:1122:3344:101::1]:56984 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
1932024-07-19T17:44:19.073ZINFOSledAgent (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
1942024-07-19T17:44:19.074ZINFOSledAgent (producer-server): starting oximeter metric producer server address = [fd00:1122:3344:101::1]:56984 file = oximeter/producer/src/lib.rs:283 interval = 30s producer_id = 0c1108fb-cc91-4535-adf7-846d50ef8716 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
1952024-07-19T17:44:19.075ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 203.361139ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
1962024-07-19T17:44:19.119ZINFOSledAgent (VmmReservoirManager): 50% of 102952026112 physical ram = 51476013056 bytes) file = sled-agent/src/vmm_reservoir.rs:267 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
1972024-07-19T17:44:19.119ZINFOSledAgent (VmmReservoirManager): Setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:275 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
1982024-07-19T17:44:19.119ZINFOSledAgent (ServiceManager): sled agent started file = sled-agent/src/services.rs:905 underlay_address = fd00:1122:3344:101::1
1992024-07-19T17:44:19.119ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
2002024-07-19T17:44:19.120ZINFOSledAgent (dropshot (SledAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:205 local_addr = [fd00:1122:3344:101::1]:12345
2012024-07-19T17:44:19.120ZINFOSledAgent (SledAgentMain): Sled Agent loaded; recording configuration file = sled-agent/src/bootstrap/server.rs:430
2022024-07-19T17:44:19.120ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
2032024-07-19T17:44:19.120ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2042024-07-19T17:44:19.120ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2052024-07-19T17:44:19.120ZINFOSledAgent (HardwareMonitor): Sled Agent Started file = sled-agent/src/hardware_monitor.rs:138
2062024-07-19T17:44:19.120ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = Some((fd00:1122:3344:101::2, Some(RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56 }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [PortConfigV2 { routes: [RouteConfig { destination: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), nexthop: 10.151.1.100, vlan_id: None }], addresses: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false }], bgp: [], bfd: [] })))
2072024-07-19T17:44:19.120ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3583
2082024-07-19T17:44:19.120ZINFOSledAgent (SledAgentSprocketsServer): Connection closed file = sled-agent/src/bootstrap/sprockets_server.rs:73 remote_addr = [fdb0:18c0:4d0d:9fb2::1]:60684
2092024-07-19T17:44:19.318ZINFOSledAgent (BootstrapAgentRssHandler): Peer agent initialized file = sled-agent/src/bootstrap/rss_handle.rs:108 peer = [fdb0:18c0:4d0d:9fb2::1]:12346
2102024-07-19T17:44:19.318ZINFOSledAgent (BootstrapAgentRssHandler): Initialized sled agent file = sled-agent/src/bootstrap/rss_handle.rs:232 target_sled = [fdb0:18c0:4d0d:9fb2::1]:12346
2112024-07-19T17:44:19.319ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
2122024-07-19T17:44:19.319ZINFOSledAgent (RSS): No ledger in /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/rss-service-plan-v3.json file = common/src/ledger.rs:188
2132024-07-19T17:44:19.319ZINFOSledAgent (RSS): No ledger in /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/rss-service-plan-v3.json file = common/src/ledger.rs:188
2142024-07-19T17:44:19.319ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
2152024-07-19T17:44:19.319ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
2162024-07-19T17:44:19.320ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 732.114598ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
2172024-07-19T17:44:19.325ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/0c1108fb-cc91-4535-adf7-846d50ef8716): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
2182024-07-19T17:44:19.351ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:64204
2192024-07-19T17:44:19.351ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
2202024-07-19T17:44:19.351ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 169 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:64204 req_id = 01e73c61-a89c-4d5d-bfd4-1513a9fc6a3d response_code = 200 uri = /inventory
2212024-07-19T17:44:19.380ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:57380
2222024-07-19T17:44:19.380ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 43 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:57380 req_id = 9bbb54d6-e162-4d79-b430-5b004da1e125 response_code = 200 uri = /sled-role
2232024-07-19T17:44:19.381ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
2242024-07-19T17:44:19.381ZINFOSledAgent (RSS): Writing ledger to /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/.rss-service-plan-v3.json.tmp file = common/src/ledger.rs:199
2252024-07-19T17:44:19.381ZINFOSledAgent (RSS): Writing ledger to /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/.rss-service-plan-v3.json.tmp file = common/src/ledger.rs:199
2262024-07-19T17:44:19.382ZINFOSledAgent (RSS): Service plan written to storage file = sled-agent/src/rack_setup/plan/service.rs:832
2272024-07-19T17:44:19.410ZINFOSledAgent (RSS): attempting to set up sled's storage: OmicronPhysicalDisksConfig { disks: [OmicronPhysicalDiskConfig { id: b569542b-5f11-4b90-84d5-387811c14788, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, pool_id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool) }, OmicronPhysicalDiskConfig { id: b06cbc3e-b215-4c8e-aed8-166fadbae657, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, pool_id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool) }, OmicronPhysicalDiskConfig { id: 2978d5b5-aa61-4523-889e-0407173d3a4e, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, pool_id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool) }, OmicronPhysicalDiskConfig { id: 593121ba-1210-4f7a-b04a-8c7ecd0ef1f9, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, pool_id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool) }, OmicronPhysicalDiskConfig { id: 5939261f-cd52-4755-bd75-57dec66dea03, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, pool_id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool) }, OmicronPhysicalDiskConfig { id: 039410aa-de10-4420-b346-3db59201d69b, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, pool_id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool) }, OmicronPhysicalDiskConfig { id: ebc8c4cf-db7e-4645-b934-415332d8354e, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, pool_id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool) }, OmicronPhysicalDiskConfig { id: 76c9fa56-82f8-4131-a2ba-1b9a8c208bf2, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, pool_id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool) }, OmicronPhysicalDiskConfig { id: 6eb91952-c334-44c1-807e-728f7748b534, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, pool_id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool) }], generation: Generation(1) } file = sled-agent/src/rack_setup/service.rs:348 sled_address = [fd00:1122:3344:101::1]:12345
2282024-07-19T17:44:19.410ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:49963
2292024-07-19T17:44:19.411ZINFOSledAgent: physical disks ensure file = sled-agent/src/sled_agent.rs:825 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
2302024-07-19T17:44:19.411ZINFOSledAgent (StorageManager): Received OmicronPhysicalDisksEnsure { config: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: b569542b-5f11-4b90-84d5-387811c14788, pool_id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: b06cbc3e-b215-4c8e-aed8-166fadbae657, pool_id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: 2978d5b5-aa61-4523-889e-0407173d3a4e, pool_id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: 593121ba-1210-4f7a-b04a-8c7ecd0ef1f9, pool_id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: 5939261f-cd52-4755-bd75-57dec66dea03, pool_id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: 039410aa-de10-4420-b346-3db59201d69b, pool_id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: ebc8c4cf-db7e-4645-b934-415332d8354e, pool_id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: 76c9fa56-82f8-4131-a2ba-1b9a8c208bf2, pool_id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: 6eb91952-c334-44c1-807e-728f7748b534, pool_id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool) }] }, tx: ... } file = sled-storage/src/manager.rs:404
2312024-07-19T17:44:19.411ZINFOSledAgent (StorageManager): No ledger in /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2322024-07-19T17:44:19.411ZINFOSledAgent (StorageManager): No ledger in /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2332024-07-19T17:44:19.411ZINFOSledAgent (StorageManager): No previously-stored 'requested disks', creating new ledger file = sled-storage/src/manager.rs:643 request = omicron_physical_disks_ensure
2342024-07-19T17:44:19.411ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:374
2352024-07-19T17:44:19.411ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:414
2362024-07-19T17:44:19.411ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
2372024-07-19T17:44:19.411ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:127 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } path = /scratch/u2_0.vdev
2382024-07-19T17:44:19.533ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_0.vdev file = sled-hardware/src/disk.rs:361
2392024-07-19T17:44:19.533ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:367 id = 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool)
2402024-07-19T17:44:20.191ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 785.593685ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
2412024-07-19T17:44:20.191ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/0c1108fb-cc91-4535-adf7-846d50ef8716): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
2422024-07-19T17:44:20.611ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }
2432024-07-19T17:44:20.808ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:332
2442024-07-19T17:44:20.808ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:334
2452024-07-19T17:44:20.808ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:338
2462024-07-19T17:44:20.808ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:340
2472024-07-19T17:44:20.808ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2482024-07-19T17:44:20.808ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2492024-07-19T17:44:20.808ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
2502024-07-19T17:44:21.030ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 1.415004669s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
2512024-07-19T17:44:21.230ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:465 zone = oxz_switch
2522024-07-19T17:44:21.921ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/0c1108fb-cc91-4535-adf7-846d50ef8716): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
2532024-07-19T17:44:21.921ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/0c1108fb-cc91-4535-adf7-846d50ef8716): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
2542024-07-19T17:44:21.921ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2552024-07-19T17:44:22.137ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 50.769637ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2562024-07-19T17:44:22.151ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_988e775a-ddc4-455d-bc4e-f2e3c228cc74/crypt/zone file = sled-storage/src/dataset.rs:395
2572024-07-19T17:44:22.755ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 125.633507ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2582024-07-19T17:44:22.756ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 5.870274815s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
2592024-07-19T17:44:23.205ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/0c1108fb-cc91-4535-adf7-846d50ef8716): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
2602024-07-19T17:44:23.397ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 202.191772ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2612024-07-19T17:44:24.000ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 421.610628ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2622024-07-19T17:44:24.002ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/0c1108fb-cc91-4535-adf7-846d50ef8716): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
2632024-07-19T17:44:24.207ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/0c1108fb-cc91-4535-adf7-846d50ef8716): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
2642024-07-19T17:44:25.075ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 803.068703ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2652024-07-19T17:44:25.088ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_988e775a-ddc4-455d-bc4e-f2e3c228cc74 file = sled-storage/src/dataset.rs:480
2662024-07-19T17:44:25.284ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/0c1108fb-cc91-4535-adf7-846d50ef8716): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
2672024-07-19T17:44:25.290ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:512
2682024-07-19T17:44:25.290ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:429
2692024-07-19T17:44:25.290ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:414
2702024-07-19T17:44:25.290ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
2712024-07-19T17:44:25.290ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:127 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } path = /scratch/u2_1.vdev
2722024-07-19T17:44:25.506ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_1.vdev file = sled-hardware/src/disk.rs:361
2732024-07-19T17:44:25.506ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:367 id = f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool)
2742024-07-19T17:44:26.582ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }
2752024-07-19T17:44:26.804ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:332
2762024-07-19T17:44:26.804ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:334
2772024-07-19T17:44:26.804ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:338
2782024-07-19T17:44:26.804ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:340
2792024-07-19T17:44:26.804ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2802024-07-19T17:44:26.804ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2812024-07-19T17:44:26.804ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
2822024-07-19T17:44:27.464ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/0c1108fb-cc91-4535-adf7-846d50ef8716): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
2832024-07-19T17:44:27.464ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/0c1108fb-cc91-4535-adf7-846d50ef8716): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
2842024-07-19T17:44:28.132ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2852024-07-19T17:44:28.602ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_f3be927c-0009-45ce-8f85-0c94f9b67b8f/crypt/zone file = sled-storage/src/dataset.rs:395
2862024-07-19T17:44:29.081ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 7.012443366s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
2872024-07-19T17:44:29.309ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/0c1108fb-cc91-4535-adf7-846d50ef8716): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
2882024-07-19T17:44:29.309ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/0c1108fb-cc91-4535-adf7-846d50ef8716): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
2892024-07-19T17:44:29.516ZINFOSledAgent (VmmReservoirManager): Finished setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:279 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
2902024-07-19T17:44:29.623ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_f3be927c-0009-45ce-8f85-0c94f9b67b8f file = sled-storage/src/dataset.rs:480
2912024-07-19T17:44:29.632ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:512
2922024-07-19T17:44:29.632ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:429
2932024-07-19T17:44:29.632ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:414
2942024-07-19T17:44:29.632ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
2952024-07-19T17:44:29.632ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:127 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } path = /scratch/u2_2.vdev
2962024-07-19T17:44:29.641ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_2.vdev file = sled-hardware/src/disk.rs:361
2972024-07-19T17:44:29.641ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:367 id = ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool)
2982024-07-19T17:44:29.735ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }
2992024-07-19T17:44:29.743ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:332
3002024-07-19T17:44:29.743ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:334
3012024-07-19T17:44:29.743ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:338
3022024-07-19T17:44:29.743ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:340
3032024-07-19T17:44:29.743ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3042024-07-19T17:44:29.743ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3052024-07-19T17:44:29.743ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3062024-07-19T17:44:29.779ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3072024-07-19T17:44:29.787ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765/crypt/zone file = sled-storage/src/dataset.rs:395
3082024-07-19T17:44:29.910ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 file = sled-storage/src/dataset.rs:480
3092024-07-19T17:44:29.918ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:512
3102024-07-19T17:44:29.918ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:429
3112024-07-19T17:44:29.918ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:414
3122024-07-19T17:44:29.918ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
3132024-07-19T17:44:29.918ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:127 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } path = /scratch/u2_3.vdev
3142024-07-19T17:44:29.927ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_3.vdev file = sled-hardware/src/disk.rs:361
3152024-07-19T17:44:29.927ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:367 id = 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool)
3162024-07-19T17:44:30.020ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }
3172024-07-19T17:44:30.029ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:332
3182024-07-19T17:44:30.029ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:334
3192024-07-19T17:44:30.029ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:338
3202024-07-19T17:44:30.029ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:340
3212024-07-19T17:44:30.029ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3222024-07-19T17:44:30.029ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3232024-07-19T17:44:30.029ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3242024-07-19T17:44:30.066ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3252024-07-19T17:44:30.075ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_4c50992d-c53d-42b7-a46f-5340e3aee556/crypt/zone file = sled-storage/src/dataset.rs:395
3262024-07-19T17:44:30.200ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_4c50992d-c53d-42b7-a46f-5340e3aee556 file = sled-storage/src/dataset.rs:480
3272024-07-19T17:44:30.208ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:512
3282024-07-19T17:44:30.208ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:429
3292024-07-19T17:44:30.208ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:414
3302024-07-19T17:44:30.208ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
3312024-07-19T17:44:30.208ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:127 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } path = /scratch/u2_4.vdev
3322024-07-19T17:44:30.218ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_4.vdev file = sled-hardware/src/disk.rs:361
3332024-07-19T17:44:30.218ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:367 id = 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool)
3342024-07-19T17:44:30.310ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }
3352024-07-19T17:44:30.318ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:332
3362024-07-19T17:44:30.318ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:334
3372024-07-19T17:44:30.318ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:338
3382024-07-19T17:44:30.318ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:340
3392024-07-19T17:44:30.318ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3402024-07-19T17:44:30.318ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3412024-07-19T17:44:30.318ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3422024-07-19T17:44:30.323ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/0c1108fb-cc91-4535-adf7-846d50ef8716): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
3432024-07-19T17:44:30.354ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3442024-07-19T17:44:30.362ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_7c02949e-12b1-4d82-9ac3-2efd807aef95/crypt/zone file = sled-storage/src/dataset.rs:395
3452024-07-19T17:44:30.484ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_7c02949e-12b1-4d82-9ac3-2efd807aef95 file = sled-storage/src/dataset.rs:480
3462024-07-19T17:44:30.491ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:512
3472024-07-19T17:44:30.492ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:429
3482024-07-19T17:44:30.492ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:414
3492024-07-19T17:44:30.492ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
3502024-07-19T17:44:30.492ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:127 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } path = /scratch/u2_5.vdev
3512024-07-19T17:44:30.501ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_5.vdev file = sled-hardware/src/disk.rs:361
3522024-07-19T17:44:30.501ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:367 id = 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool)
3532024-07-19T17:44:30.593ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }
3542024-07-19T17:44:30.600ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:332
3552024-07-19T17:44:30.600ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:334
3562024-07-19T17:44:30.600ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:338
3572024-07-19T17:44:30.600ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:340
3582024-07-19T17:44:30.600ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3592024-07-19T17:44:30.600ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3602024-07-19T17:44:30.600ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3612024-07-19T17:44:30.632ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3622024-07-19T17:44:30.640ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_765bc25e-72d0-4d30-8cfe-a58b7db5c2e8/crypt/zone file = sled-storage/src/dataset.rs:395
3632024-07-19T17:44:30.758ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 file = sled-storage/src/dataset.rs:480
3642024-07-19T17:44:30.765ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:512
3652024-07-19T17:44:30.765ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:429
3662024-07-19T17:44:30.765ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:414
3672024-07-19T17:44:30.765ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
3682024-07-19T17:44:30.765ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:127 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } path = /scratch/u2_6.vdev
3692024-07-19T17:44:30.773ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_6.vdev file = sled-hardware/src/disk.rs:361
3702024-07-19T17:44:30.773ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:367 id = 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool)
3712024-07-19T17:44:30.869ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }
3722024-07-19T17:44:30.877ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:332
3732024-07-19T17:44:30.877ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:334
3742024-07-19T17:44:30.877ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:338
3752024-07-19T17:44:30.877ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:340
3762024-07-19T17:44:30.877ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3772024-07-19T17:44:30.877ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3782024-07-19T17:44:30.877ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3792024-07-19T17:44:30.880ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/0c1108fb-cc91-4535-adf7-846d50ef8716): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
3802024-07-19T17:44:30.908ZINFOSledAgent (ServiceManager): Ensuring tfportrear0_0/ll exists in zone file = sled-agent/src/services.rs:2256
3812024-07-19T17:44:30.918ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3822024-07-19T17:44:30.925ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_696d009b-6d0c-4082-b81b-231c6c85d568/crypt/zone file = sled-storage/src/dataset.rs:395
3832024-07-19T17:44:31.054ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_696d009b-6d0c-4082-b81b-231c6c85d568 file = sled-storage/src/dataset.rs:480
3842024-07-19T17:44:31.062ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:512
3852024-07-19T17:44:31.062ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:429
3862024-07-19T17:44:31.062ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:414
3872024-07-19T17:44:31.062ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
3882024-07-19T17:44:31.062ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:127 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } path = /scratch/u2_7.vdev
3892024-07-19T17:44:31.072ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_7.vdev file = sled-hardware/src/disk.rs:361
3902024-07-19T17:44:31.072ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:367 id = 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool)
3912024-07-19T17:44:31.167ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }
3922024-07-19T17:44:31.175ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:332
3932024-07-19T17:44:31.175ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:334
3942024-07-19T17:44:31.175ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:338
3952024-07-19T17:44:31.175ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:340
3962024-07-19T17:44:31.175ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
3972024-07-19T17:44:31.176ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
3982024-07-19T17:44:31.176ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3992024-07-19T17:44:31.213ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4002024-07-19T17:44:31.220ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_311acf6c-db4a-40cb-bd1f-e33d2bc8747f/crypt/zone file = sled-storage/src/dataset.rs:395
4012024-07-19T17:44:31.324ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/0c1108fb-cc91-4535-adf7-846d50ef8716): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
4022024-07-19T17:44:31.341ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_311acf6c-db4a-40cb-bd1f-e33d2bc8747f file = sled-storage/src/dataset.rs:480
4032024-07-19T17:44:31.349ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:512
4042024-07-19T17:44:31.349ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:429
4052024-07-19T17:44:31.349ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:414
4062024-07-19T17:44:31.349ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
4072024-07-19T17:44:31.349ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:127 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } path = /scratch/u2_8.vdev
4082024-07-19T17:44:31.359ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_8.vdev file = sled-hardware/src/disk.rs:361
4092024-07-19T17:44:31.359ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:367 id = 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool)
4102024-07-19T17:44:31.449ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }
4112024-07-19T17:44:31.456ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:332
4122024-07-19T17:44:31.456ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:334
4132024-07-19T17:44:31.456ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:338
4142024-07-19T17:44:31.456ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:340
4152024-07-19T17:44:31.456ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4162024-07-19T17:44:31.457ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4172024-07-19T17:44:31.457ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
4182024-07-19T17:44:31.490ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4192024-07-19T17:44:31.498ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_41faf794-cdf9-49a0-b529-4c18e335c28c/crypt/zone file = sled-storage/src/dataset.rs:395
4202024-07-19T17:44:31.624ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_41faf794-cdf9-49a0-b529-4c18e335c28c file = sled-storage/src/dataset.rs:480
4212024-07-19T17:44:31.633ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:512
4222024-07-19T17:44:31.633ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:429
4232024-07-19T17:44:31.633ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4242024-07-19T17:44:31.633ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 301424fa-c483-4ce9-92fb-3c8ee10e6f9f (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 20dccfdd-e68a-468d-8629-7f7e0834aa1c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
4252024-07-19T17:44:31.633ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
4262024-07-19T17:44:31.633ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
4272024-07-19T17:44:31.633ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4282024-07-19T17:44:31.634ZINFOSledAgent: physical disks ensure: Updated storage file = sled-agent/src/sled_agent.rs:829 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
4292024-07-19T17:44:31.634ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
4302024-07-19T17:44:31.634ZINFOSledAgent: physical disks ensure: Propagating new generation of disks file = sled-agent/src/sled_agent.rs:855 generation = Generation(1) sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
4312024-07-19T17:44:31.634ZINFOSledAgent: physical disks ensure: Updated storage monitor file = sled-agent/src/sled_agent.rs:860 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
4322024-07-19T17:44:31.634ZINFOSledAgent: physical disks ensure: Updated zone bundler file = sled-agent/src/sled_agent.rs:865 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
4332024-07-19T17:44:31.634ZINFOSledAgent (ProbeManager): use_only_these_disks: Processing new request file = sled-agent/src/probe_manager.rs:116 gen = Generation(1) sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
4342024-07-19T17:44:31.634ZINFOSledAgent: physical disks ensure: Updated probes file = sled-agent/src/sled_agent.rs:871 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
4352024-07-19T17:44:31.634ZINFOSledAgent (InstanceManager): use_only_these_disks: Processing new request file = sled-agent/src/instance_manager.rs:822 gen = Generation(1)
4362024-07-19T17:44:31.634ZINFOSledAgent: physical disks ensure: Updated instances file = sled-agent/src/sled_agent.rs:876 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
4372024-07-19T17:44:31.634ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 12223353 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49963 req_id = 5719d743-a8d4-4691-861c-93baccb007f0 response_code = 200 uri = /omicron-physical-disks
4382024-07-19T17:44:31.665ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(2), zones: [OmicronZoneConfig { id: 8ae14f97-bd01-42ad-87d5-23eae9194521, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
4392024-07-19T17:44:31.665ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:38265
4402024-07-19T17:44:31.665ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
4412024-07-19T17:44:31.665ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4422024-07-19T17:44:31.716ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
4432024-07-19T17:44:31.716ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }
4442024-07-19T17:44:31.766ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
4452024-07-19T17:44:31.767ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4462024-07-19T17:44:31.817ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
4472024-07-19T17:44:31.817ZINFOSledAgent (ServiceManager): No ledger in /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/omicron-zones.json file = common/src/ledger.rs:188
4482024-07-19T17:44:31.817ZINFOSledAgent (ServiceManager): No ledger in /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/omicron-zones.json file = common/src/ledger.rs:188
4492024-07-19T17:44:31.817ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
4502024-07-19T17:44:31.862ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
4512024-07-19T17:44:31.862ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
4522024-07-19T17:44:31.862ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
4532024-07-19T17:44:31.914ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_98b939a3-2514-4766-b854-3b3ddf8b23f9 file = illumos-utils/src/zone.rs:316
4542024-07-19T17:44:31.918ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_8f9c564d-5823-4b85-a0ff-2de7f54b602e file = illumos-utils/src/zone.rs:316
4552024-07-19T17:44:31.922ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_8ae14f97-bd01-42ad-87d5-23eae9194521 file = illumos-utils/src/zone.rs:316
4562024-07-19T17:44:31.934ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_98b939a3-2514-4766-b854-3b3ddf8b23f9 file = illumos-utils/src/zone.rs:354
4572024-07-19T17:44:31.938ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_8f9c564d-5823-4b85-a0ff-2de7f54b602e file = illumos-utils/src/zone.rs:354
4582024-07-19T17:44:31.942ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_8ae14f97-bd01-42ad-87d5-23eae9194521 file = illumos-utils/src/zone.rs:354
4592024-07-19T17:44:32.326ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/0c1108fb-cc91-4535-adf7-846d50ef8716): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
4602024-07-19T17:44:32.879ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/0c1108fb-cc91-4535-adf7-846d50ef8716): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
4612024-07-19T17:44:33.456ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_8ae14f97-bd01-42ad-87d5-23eae9194521: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService3"/> <propval type="astring" name="gateway" value="fd00:1122:3344:1::2"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:1::1"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:1::1]:5353"/> <propval type="astring" name="dns_address" value="[fd00:1122:3344:1::1]:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4622024-07-19T17:44:33.456ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:1::, width: 64 }
4632024-07-19T17:44:33.456ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:465 zone = oxz_internal_dns_8ae14f97-bd01-42ad-87d5-23eae9194521
4642024-07-19T17:44:33.943ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_98b939a3-2514-4766-b854-3b3ddf8b23f9: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService1"/> <propval type="astring" name="gateway" value="fd00:1122:3344:2::2"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:2::1"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:2::1]:5353"/> <propval type="astring" name="dns_address" value="[fd00:1122:3344:2::1]:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4652024-07-19T17:44:34.422ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:2::, width: 64 }
4662024-07-19T17:44:34.422ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_8f9c564d-5823-4b85-a0ff-2de7f54b602e: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService2"/> <propval type="astring" name="gateway" value="fd00:1122:3344:3::2"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:3::1"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:3::1]:5353"/> <propval type="astring" name="dns_address" value="[fd00:1122:3344:3::1]:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4672024-07-19T17:44:34.429ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8ae14f97-bd01-42ad-87d5-23eae9194521") failed: Property not found. retry in 51.214154ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8ae14f97-bd01-42ad-87d5-23eae9194521
4682024-07-19T17:44:34.429ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:3::, width: 64 }
4692024-07-19T17:44:34.429ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:465 zone = oxz_internal_dns_98b939a3-2514-4766-b854-3b3ddf8b23f9
4702024-07-19T17:44:34.433ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:465 zone = oxz_internal_dns_8f9c564d-5823-4b85-a0ff-2de7f54b602e
4712024-07-19T17:44:34.489ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8ae14f97-bd01-42ad-87d5-23eae9194521") failed: Property not found. retry in 62.81744ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8ae14f97-bd01-42ad-87d5-23eae9194521
4722024-07-19T17:44:34.556ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_98b939a3-2514-4766-b854-3b3ddf8b23f9") failed: Property not found. retry in 49.494093ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_98b939a3-2514-4766-b854-3b3ddf8b23f9
4732024-07-19T17:44:34.562ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8f9c564d-5823-4b85-a0ff-2de7f54b602e") failed: Property not found. retry in 39.593444ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8f9c564d-5823-4b85-a0ff-2de7f54b602e
4742024-07-19T17:44:34.569ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8ae14f97-bd01-42ad-87d5-23eae9194521") failed: Property not found. retry in 117.421629ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8ae14f97-bd01-42ad-87d5-23eae9194521
4752024-07-19T17:44:34.608ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8f9c564d-5823-4b85-a0ff-2de7f54b602e") failed: Property not found. retry in 53.914648ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8f9c564d-5823-4b85-a0ff-2de7f54b602e
4762024-07-19T17:44:34.614ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_98b939a3-2514-4766-b854-3b3ddf8b23f9") failed: Property not found. retry in 122.57237ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_98b939a3-2514-4766-b854-3b3ddf8b23f9
4772024-07-19T17:44:34.707ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8f9c564d-5823-4b85-a0ff-2de7f54b602e") failed: Property not found. retry in 235.381217ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8f9c564d-5823-4b85-a0ff-2de7f54b602e
4782024-07-19T17:44:34.729ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8ae14f97-bd01-42ad-87d5-23eae9194521") failed: Property not found. retry in 410.878053ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8ae14f97-bd01-42ad-87d5-23eae9194521
4792024-07-19T17:44:34.751ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_98b939a3-2514-4766-b854-3b3ddf8b23f9") failed: Property not found. retry in 117.60975ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_98b939a3-2514-4766-b854-3b3ddf8b23f9
4802024-07-19T17:44:34.896ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_98b939a3-2514-4766-b854-3b3ddf8b23f9") failed: Property not found. retry in 461.270286ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_98b939a3-2514-4766-b854-3b3ddf8b23f9
4812024-07-19T17:44:34.960ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8f9c564d-5823-4b85-a0ff-2de7f54b602e") failed: Property not found. retry in 224.149966ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8f9c564d-5823-4b85-a0ff-2de7f54b602e
4822024-07-19T17:44:35.150ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8ae14f97-bd01-42ad-87d5-23eae9194521") failed: Property not found. retry in 576.269057ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8ae14f97-bd01-42ad-87d5-23eae9194521
4832024-07-19T17:44:35.195ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8f9c564d-5823-4b85-a0ff-2de7f54b602e") failed: Property not found. retry in 726.229593ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8f9c564d-5823-4b85-a0ff-2de7f54b602e
4842024-07-19T17:44:35.366ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_98b939a3-2514-4766-b854-3b3ddf8b23f9") failed: Property not found. retry in 717.225762ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_98b939a3-2514-4766-b854-3b3ddf8b23f9
4852024-07-19T17:44:35.736ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8ae14f97-bd01-42ad-87d5-23eae9194521") failed: Property not found. retry in 1.24587099s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8ae14f97-bd01-42ad-87d5-23eae9194521
4862024-07-19T17:44:35.932ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8f9c564d-5823-4b85-a0ff-2de7f54b602e") failed: Property not found. retry in 1.45186029s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8f9c564d-5823-4b85-a0ff-2de7f54b602e
4872024-07-19T17:44:36.093ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_98b939a3-2514-4766-b854-3b3ddf8b23f9") failed: Property not found. retry in 539.821822ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_98b939a3-2514-4766-b854-3b3ddf8b23f9
4882024-07-19T17:44:36.641ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_98b939a3-2514-4766-b854-3b3ddf8b23f9") failed: Property not found. retry in 936.663938ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_98b939a3-2514-4766-b854-3b3ddf8b23f9
4892024-07-19T17:44:36.963ZINFOSledAgent (ServiceManager): Ensuring bootstrap address fdb0:18c0:4d0d:9fb2::2 exists in switch zone file = sled-agent/src/services.rs:2273
4902024-07-19T17:44:36.963ZINFOSledAgent (ServiceManager): Adding bootstrap address file = illumos-utils/src/running_zone.rs:578 zone = oxz_switch
4912024-07-19T17:44:36.990ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8ae14f97-bd01-42ad-87d5-23eae9194521") failed: Property not found. retry in 1.335452263s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8ae14f97-bd01-42ad-87d5-23eae9194521
4922024-07-19T17:44:37.392ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8f9c564d-5823-4b85-a0ff-2de7f54b602e") failed: Property not found. retry in 665.627697ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8f9c564d-5823-4b85-a0ff-2de7f54b602e
4932024-07-19T17:44:37.507ZINFOSledAgent (ServiceManager): Forwarding bootstrap traffic via oxBootstrap0 to fe80::8:20ff:fea8:ad0b file = sled-agent/src/services.rs:2280
4942024-07-19T17:44:37.512ZINFOSledAgent (ServiceManager): Zone not using gateway (even though underlay is up) file = sled-agent/src/services.rs:2341
4952024-07-19T17:44:37.587ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_98b939a3-2514-4766-b854-3b3ddf8b23f9") failed: Property not found. retry in 1.46205302s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_98b939a3-2514-4766-b854-3b3ddf8b23f9
4962024-07-19T17:44:37.635ZINFOSledAgent (ServiceManager): Setting up dendrite service file = sled-agent/src/services.rs:2526
4972024-07-19T17:44:37.876ZINFOSledAgent (ServiceManager): Setting up lldpd service file = sled-agent/src/services.rs:2690
4982024-07-19T17:44:38.062ZINFOSledAgent (ServiceManager): Setting up MGS service file = sled-agent/src/services.rs:2405
4992024-07-19T17:44:38.067ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8f9c564d-5823-4b85-a0ff-2de7f54b602e") failed: Property not found. retry in 883.095783ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8f9c564d-5823-4b85-a0ff-2de7f54b602e
5002024-07-19T17:44:38.335ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8ae14f97-bd01-42ad-87d5-23eae9194521") failed: Property not found. retry in 1.111038184s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8ae14f97-bd01-42ad-87d5-23eae9194521
5012024-07-19T17:44:38.364ZINFOSledAgent (ServiceManager): Setting up wicketd service file = sled-agent/src/services.rs:2440
5022024-07-19T17:44:38.583ZINFOSledAgent (ServiceManager): Setting up mgd service file = sled-agent/src/services.rs:2740
5032024-07-19T17:44:38.770ZINFOSledAgent (ServiceManager): Setting up mg-ddm service file = sled-agent/src/services.rs:2767
5042024-07-19T17:44:38.960ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8f9c564d-5823-4b85-a0ff-2de7f54b602e") failed: Property not found. retry in 574.607279ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8f9c564d-5823-4b85-a0ff-2de7f54b602e
5052024-07-19T17:44:38.987ZINFOSledAgent (ServiceManager): Setting up tfport service file = sled-agent/src/services.rs:2634
5062024-07-19T17:44:39.059ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_98b939a3-2514-4766-b854-3b3ddf8b23f9") failed: Property not found. retry in 1.486338133s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_98b939a3-2514-4766-b854-3b3ddf8b23f9
5072024-07-19T17:44:39.173ZINFOSledAgent (ServiceManager): Setting up Simulated SP service file = sled-agent/src/services.rs:2434
5082024-07-19T17:44:39.183ZINFOSledAgent (ServiceManager): Re-enabling running switch zone (new address) file = sled-agent/src/services.rs:3839 new = [fd00:1122:3344:101::2, ::1] old = [::1]
5092024-07-19T17:44:39.183ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists file = sled-agent/src/services.rs:3855
5102024-07-19T17:44:39.183ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::2, prefix: 64 })) file = illumos-utils/src/running_zone.rs:561 zone = oxz_switch
5112024-07-19T17:44:39.741ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists - OK file = sled-agent/src/services.rs:3863
5122024-07-19T17:44:39.748ZINFOSledAgent (ServiceManager): configuring dendrite service file = sled-agent/src/services.rs:3919
5132024-07-19T17:44:39.892ZINFOSledAgent (ServiceManager): configuring lldp service file = sled-agent/src/services.rs:3981
5142024-07-19T17:44:39.991ZINFOSledAgent (ServiceManager): configuring wicketd file = sled-agent/src/services.rs:3962 rack_subnet = fd00:1122:3344::
5152024-07-19T17:44:40.016ZINFOSledAgent (ServiceManager): configuring mgd service file = sled-agent/src/services.rs:4009
5162024-07-19T17:44:40.109ZINFOSledAgent (ServiceManager): configuring mg-ddm service file = sled-agent/src/services.rs:4036
5172024-07-19T17:44:40.228ZINFOSledAgent (ServiceManager): Determining physical location of our switch zone at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:369
5182024-07-19T17:44:40.260ZINFOSledAgent (ServiceManager): Initializing 1 Uplinks on Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:417
5192024-07-19T17:44:40.293ZINFOSledAgent (ServiceManager): Building Port Configuration file = sled-agent/src/bootstrap/early_networking.rs:656
5202024-07-19T17:44:40.293ZINFOSledAgent (ServiceManager): Checking dendrite uptime file = sled-agent/src/bootstrap/early_networking.rs:694
5212024-07-19T17:44:40.293ZINFOSledAgent (ServiceManager): Dendrite online file = sled-agent/src/bootstrap/early_networking.rs:697 uptime = 3
5222024-07-19T17:44:40.293ZINFOSledAgent (ServiceManager): Configuring default uplink on switch config = PortSettings {\n links: {\n "0": LinkSettings {\n addrs: [\n 10.151.1.174,\n ],\n params: LinkCreate {\n autoneg: false,\n fec: None,\n kr: false,\n lane: Some(\n LinkId(\n 0,\n ),\n ),\n speed: Speed40G,\n },\n },\n },\n} file = sled-agent/src/bootstrap/early_networking.rs:439
5232024-07-19T17:44:40.370ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:404
5242024-07-19T17:44:40.370ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5252024-07-19T17:44:40.370ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5262024-07-19T17:44:40.370ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5272024-07-19T17:44:40.370ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5282024-07-19T17:44:40.370ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5292024-07-19T17:44:40.370ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5302024-07-19T17:44:40.370ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5312024-07-19T17:44:40.370ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5322024-07-19T17:44:40.370ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5332024-07-19T17:44:40.370ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5342024-07-19T17:44:40.370ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5352024-07-19T17:44:40.370ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5362024-07-19T17:44:40.370ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5372024-07-19T17:44:40.370ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5382024-07-19T17:44:40.370ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5392024-07-19T17:44:40.370ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5402024-07-19T17:44:40.370ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5412024-07-19T17:44:40.370ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5422024-07-19T17:44:40.370ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5432024-07-19T17:44:40.370ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5442024-07-19T17:44:40.370ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5452024-07-19T17:44:40.370ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5462024-07-19T17:44:40.566ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2980 zone = oxz_internal_dns_98b939a3-2514-4766-b854-3b3ddf8b23f9
5472024-07-19T17:44:40.566ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2980 zone = oxz_internal_dns_8f9c564d-5823-4b85-a0ff-2de7f54b602e
5482024-07-19T17:44:40.566ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2980 zone = oxz_internal_dns_8ae14f97-bd01-42ad-87d5-23eae9194521
5492024-07-19T17:44:40.566ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5502024-07-19T17:44:40.566ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5512024-07-19T17:44:40.566ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 8901562 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:38265 req_id = e36d444e-2d9d-4629-bb6e-d299c450aefa response_code = 204 uri = /omicron-zones
5522024-07-19T17:44:40.568ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:1::1]:5353 file = sled-agent/src/rack_setup/service.rs:549
5532024-07-19T17:44:40.619ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:2::1]:5353 file = sled-agent/src/rack_setup/service.rs:549
5542024-07-19T17:44:40.666ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:3::1]:5353 file = sled-agent/src/rack_setup/service.rs:549
5552024-07-19T17:44:40.711ZINFOSledAgent (RSS): Configured all DNS servers file = sled-agent/src/rack_setup/service.rs:585
5562024-07-19T17:44:40.711ZINFOSledAgent (RSS): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:263
5572024-07-19T17:44:40.743ZINFOSledAgent (RSS): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:280
5582024-07-19T17:44:40.744ZINFOSledAgent (RSS): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:305
5592024-07-19T17:44:40.744ZINFOSledAgent (RSS): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:197
5602024-07-19T17:44:40.772ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(3), zones: [OmicronZoneConfig { id: 8ae14f97-bd01-42ad-87d5-23eae9194521, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: bc60810c-eece-4cbd-a0f8-d04271924b7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bf3fc707-b452-4d30-a92b-e56b4d4122ee, kind: Service { id: bc60810c-eece-4cbd-a0f8-d04271924b7c }, name: Name("ntp-bc60810c-eece-4cbd-a0f8-d04271924b7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 196])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
5612024-07-19T17:44:40.772ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:47207
5622024-07-19T17:44:40.773ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
5632024-07-19T17:44:40.773ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5642024-07-19T17:44:40.794ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
5652024-07-19T17:44:40.794ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }
5662024-07-19T17:44:40.812ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
5672024-07-19T17:44:40.812ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5682024-07-19T17:44:40.830ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
5692024-07-19T17:44:40.830ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/omicron-zones.json file = common/src/ledger.rs:177
5702024-07-19T17:44:40.830ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/omicron-zones.json file = common/src/ledger.rs:177
5712024-07-19T17:44:40.830ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
5722024-07-19T17:44:40.841ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:263
5732024-07-19T17:44:40.873ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:280
5742024-07-19T17:44:40.874ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:305
5752024-07-19T17:44:40.874ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:197
5762024-07-19T17:44:40.903ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:426 port = Port { inner: PortInner(PortData { name: "opte0", ip: 172.30.3.5, mac: MacAddr6([168, 64, 37, 255, 128, 196]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.3.1 } }) }
5772024-07-19T17:44:40.903ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1210 zone_type = ntp
5782024-07-19T17:44:40.904ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:83
5792024-07-19T17:44:40.904ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:90
5802024-07-19T17:44:40.904ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:118
5812024-07-19T17:44:40.905ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
5822024-07-19T17:44:40.928ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_ntp_bc60810c-eece-4cbd-a0f8-d04271924b7c file = illumos-utils/src/zone.rs:316
5832024-07-19T17:44:40.946ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_ntp_bc60810c-eece-4cbd-a0f8-d04271924b7c file = illumos-utils/src/zone.rs:354
5842024-07-19T17:44:41.564ZINFOSledAgent (ServiceManager): Profile for oxz_ntp_bc60810c-eece-4cbd-a0f8-d04271924b7c: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService4"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::1b"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/chrony-setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="allow" value="fd00:1122:3344:100::/56"/> <propval type="boolean" name="boundary" value="true"/> <propval type="astring" name="server" value="0.pool.ntp.org"/> </property_group> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="1.1.1.1" /> <value_node value="9.9.9.9" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/ntp"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="opte0"/> <propval type="astring" name="gateway" value="172.30.3.1"/> <propval type="astring" name="ip" value="172.30.3.5"/> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
5852024-07-19T17:44:41.564ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:465 zone = oxz_ntp_bc60810c-eece-4cbd-a0f8-d04271924b7c
5862024-07-19T17:44:41.704ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_bc60810c-eece-4cbd-a0f8-d04271924b7c") failed: Property not found. retry in 62.163952ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_bc60810c-eece-4cbd-a0f8-d04271924b7c
5872024-07-19T17:44:41.784ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_bc60810c-eece-4cbd-a0f8-d04271924b7c") failed: Property not found. retry in 61.933489ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_bc60810c-eece-4cbd-a0f8-d04271924b7c
5882024-07-19T17:44:41.875ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_bc60810c-eece-4cbd-a0f8-d04271924b7c") failed: Property not found. retry in 119.907276ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_bc60810c-eece-4cbd-a0f8-d04271924b7c
5892024-07-19T17:44:42.006ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_bc60810c-eece-4cbd-a0f8-d04271924b7c") failed: Property not found. retry in 419.736462ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_bc60810c-eece-4cbd-a0f8-d04271924b7c
5902024-07-19T17:44:42.436ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_bc60810c-eece-4cbd-a0f8-d04271924b7c") failed: Property not found. retry in 452.468173ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_bc60810c-eece-4cbd-a0f8-d04271924b7c
5912024-07-19T17:44:42.898ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_bc60810c-eece-4cbd-a0f8-d04271924b7c") failed: Property not found. retry in 1.343545497s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_bc60810c-eece-4cbd-a0f8-d04271924b7c
5922024-07-19T17:44:43.660ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:39758
5932024-07-19T17:44:43.660ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 94 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:39758 req_id = ec2664e0-3bea-4525-93e1-a55f56e8dae1 response_code = 200 uri = /baseboard
5942024-07-19T17:44:44.251ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_bc60810c-eece-4cbd-a0f8-d04271924b7c") failed: Property not found. retry in 1.268702518s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_bc60810c-eece-4cbd-a0f8-d04271924b7c
5952024-07-19T17:44:45.530ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_bc60810c-eece-4cbd-a0f8-d04271924b7c") failed: Property not found. retry in 958.922092ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_bc60810c-eece-4cbd-a0f8-d04271924b7c
5962024-07-19T17:44:46.511ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2980 zone = oxz_ntp_bc60810c-eece-4cbd-a0f8-d04271924b7c
5972024-07-19T17:44:46.511ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5982024-07-19T17:44:46.512ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5992024-07-19T17:44:46.512ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 5739335 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:47207 req_id = c59a0d6d-0d55-4a8b-9f52-91f0e2c6b509 response_code = 204 uri = /omicron-zones
6002024-07-19T17:44:46.514ZINFOSledAgent (RSS): Waiting for rack time synchronization file = sled-agent/src/rack_setup/service.rs:626
6012024-07-19T17:44:46.544ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6022024-07-19T17:44:46.544ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:45764
6032024-07-19T17:44:46.555ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_bc60810c-eece-4cbd-a0f8-d04271924b7c': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3569
6042024-07-19T17:44:46.555ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 10651 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:45764 req_id = e0d506fb-a643-4885-8e95-1a6d48c559e9 response_code = 500 uri = /timesync
6052024-07-19T17:44:46.558ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6062024-07-19T17:44:46.651ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6072024-07-19T17:44:46.651ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:53542
6082024-07-19T17:44:50.660ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_bc60810c-eece-4cbd-a0f8-d04271924b7c': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3569
6092024-07-19T17:44:50.660ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 4008713 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:53542 req_id = 5396127a-d5ac-447d-9b0f-046bba1c466e response_code = 500 uri = /timesync
6102024-07-19T17:44:50.663ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6112024-07-19T17:44:50.835ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6122024-07-19T17:44:50.835ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42248
6132024-07-19T17:44:50.844ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_bc60810c-eece-4cbd-a0f8-d04271924b7c': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3569
6142024-07-19T17:44:50.844ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 8551 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:42248 req_id = 51dfcd2e-7c40-43a5-a9bf-10b53638892d response_code = 500 uri = /timesync
6152024-07-19T17:44:50.847ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6162024-07-19T17:44:51.085ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6172024-07-19T17:44:51.085ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:38576
6182024-07-19T17:44:51.093ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_bc60810c-eece-4cbd-a0f8-d04271924b7c': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3569
6192024-07-19T17:44:51.094ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 8448 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:38576 req_id = 52c7e2a7-bd64-4c83-887e-4e1026b34aa3 response_code = 500 uri = /timesync
6202024-07-19T17:44:51.095ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6212024-07-19T17:44:51.236ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6222024-07-19T17:44:51.236ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:41265
6232024-07-19T17:44:55.245ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 4008797 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:41265 req_id = f28f0740-d8a1-491b-8d73-cb1161949a90 response_code = 200 uri = /timesync
6242024-07-19T17:44:55.248ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:634
6252024-07-19T17:44:55.248ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6262024-07-19T17:44:55.550ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6272024-07-19T17:44:55.551ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:56295
6282024-07-19T17:44:55.560ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 8966 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:56295 req_id = f1f956ce-bf69-49e3-83b9-d13aa1d942b7 response_code = 200 uri = /timesync
6292024-07-19T17:44:55.563ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:634
6302024-07-19T17:44:55.563ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6312024-07-19T17:44:55.891ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6322024-07-19T17:44:55.891ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:35628
6332024-07-19T17:44:55.899ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 8535 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:35628 req_id = 95b8b527-1052-42f1-b80e-e81ec633122e response_code = 200 uri = /timesync
6342024-07-19T17:44:55.901ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:634
6352024-07-19T17:44:55.901ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6362024-07-19T17:44:56.103ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6372024-07-19T17:44:56.103ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42472
6382024-07-19T17:44:56.111ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 8233 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:42472 req_id = fe78d930-fa06-4b87-96b0-0493d69d3c91 response_code = 200 uri = /timesync
6392024-07-19T17:44:56.115ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:634
6402024-07-19T17:44:56.115ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6412024-07-19T17:44:56.134ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 130.376962ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
6422024-07-19T17:44:56.661ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6432024-07-19T17:44:56.661ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:33129
6442024-07-19T17:44:56.670ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 8843 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:33129 req_id = 312d7ad3-5845-4717-818c-d2efc10f3c38 response_code = 200 uri = /timesync
6452024-07-19T17:44:56.673ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:634
6462024-07-19T17:44:56.673ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6472024-07-19T17:44:56.923ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6482024-07-19T17:44:56.923ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:35152
6492024-07-19T17:44:56.931ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 7839 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:35152 req_id = a56ffd73-4675-4fe2-b3a5-d0c47ab3af2b response_code = 200 uri = /timesync
6502024-07-19T17:44:56.933ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:634
6512024-07-19T17:44:56.933ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6522024-07-19T17:44:57.364ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6532024-07-19T17:44:57.364ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:34493
6542024-07-19T17:44:57.372ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 8046 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:34493 req_id = c2604e9c-fd1b-407f-ab9d-b7f884d00c0f response_code = 200 uri = /timesync
6552024-07-19T17:44:57.375ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:634
6562024-07-19T17:44:57.375ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6572024-07-19T17:44:58.372ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6582024-07-19T17:44:58.372ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:53905
6592024-07-19T17:44:58.392ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 20567 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:53905 req_id = 15aa1b5f-a76b-4d61-b2f0-3b950683217c response_code = 200 uri = /timesync
6602024-07-19T17:44:58.395ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: true, ref_id: 1241950281, ip_addr: 74.6.168.73, stratum: 3, ref_time: 1721411097.3841624, correction: -0.0 } file = sled-agent/src/rack_setup/service.rs:634
6612024-07-19T17:44:58.395ZINFOSledAgent (RSS): Finished setting up Internal DNS and NTP file = sled-agent/src/rack_setup/service.rs:1224
6622024-07-19T17:44:58.425ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 8ae14f97-bd01-42ad-87d5-23eae9194521, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: bc60810c-eece-4cbd-a0f8-d04271924b7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bf3fc707-b452-4d30-a92b-e56b4d4122ee, kind: Service { id: bc60810c-eece-4cbd-a0f8-d04271924b7c }, name: Name("ntp-bc60810c-eece-4cbd-a0f8-d04271924b7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 196])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: c168b27b-f467-4350-a348-4006ce85a213, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 15be01e5-1765-45f0-b521-da99b395bdb3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: d8a1310d-09c1-4844-91b7-509f93b30cfd, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
6632024-07-19T17:44:58.425ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:54642
6642024-07-19T17:44:58.425ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
6652024-07-19T17:44:58.425ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6662024-07-19T17:44:58.445ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
6672024-07-19T17:44:58.445ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }
6682024-07-19T17:44:58.462ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
6692024-07-19T17:44:58.462ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6702024-07-19T17:44:58.480ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
6712024-07-19T17:44:58.480ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6722024-07-19T17:44:58.532ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
6732024-07-19T17:44:58.532ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6742024-07-19T17:44:58.583ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
6752024-07-19T17:44:58.583ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6762024-07-19T17:44:58.635ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
6772024-07-19T17:44:58.635ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6782024-07-19T17:44:58.687ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
6792024-07-19T17:44:58.687ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6802024-07-19T17:44:58.740ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
6812024-07-19T17:44:58.740ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/omicron-zones.json file = common/src/ledger.rs:177
6822024-07-19T17:44:58.740ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/omicron-zones.json file = common/src/ledger.rs:177
6832024-07-19T17:44:58.740ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
6842024-07-19T17:44:58.825ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
6852024-07-19T17:44:58.825ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
6862024-07-19T17:44:58.825ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
6872024-07-19T17:44:58.825ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
6882024-07-19T17:44:58.825ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
6892024-07-19T17:44:58.914ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0 file = illumos-utils/src/zone.rs:316
6902024-07-19T17:44:58.919ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_d8a1310d-09c1-4844-91b7-509f93b30cfd file = illumos-utils/src/zone.rs:316
6912024-07-19T17:44:58.923ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a file = illumos-utils/src/zone.rs:316
6922024-07-19T17:44:58.927ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_c168b27b-f467-4350-a348-4006ce85a213 file = illumos-utils/src/zone.rs:316
6932024-07-19T17:44:58.932ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_15be01e5-1765-45f0-b521-da99b395bdb3 file = illumos-utils/src/zone.rs:316
6942024-07-19T17:44:58.937ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0 file = illumos-utils/src/zone.rs:354
6952024-07-19T17:44:58.941ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_d8a1310d-09c1-4844-91b7-509f93b30cfd file = illumos-utils/src/zone.rs:354
6962024-07-19T17:44:58.946ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a file = illumos-utils/src/zone.rs:354
6972024-07-19T17:44:58.950ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_c168b27b-f467-4350-a348-4006ce85a213 file = illumos-utils/src/zone.rs:354
6982024-07-19T17:44:58.955ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_15be01e5-1765-45f0-b521-da99b395bdb3 file = illumos-utils/src/zone.rs:354
6992024-07-19T17:45:01.813ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_c168b27b-f467-4350-a348-4006ce85a213: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService8"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::4"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="[fd00:1122:3344:101::4]:32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value="c168b27b-f467-4350-a348-4006ce85a213"/> <propval type="astring" name="cockroach_address" value="[fd00:1122:3344:101::4]:32221"/> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::4]:32222"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7002024-07-19T17:45:01.813ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:465 zone = oxz_cockroachdb_c168b27b-f467-4350-a348-4006ce85a213
7012024-07-19T17:45:01.820ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_d8a1310d-09c1-4844-91b7-509f93b30cfd: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService6"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::7"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="[fd00:1122:3344:101::7]:32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value="d8a1310d-09c1-4844-91b7-509f93b30cfd"/> <propval type="astring" name="cockroach_address" value="[fd00:1122:3344:101::7]:32221"/> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::7]:32222"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7022024-07-19T17:45:01.820ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:465 zone = oxz_cockroachdb_d8a1310d-09c1-4844-91b7-509f93b30cfd
7032024-07-19T17:45:01.830ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService7"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::3"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="[fd00:1122:3344:101::3]:32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value="27c74baf-8069-4296-ac26-2f4818d2ab3a"/> <propval type="astring" name="cockroach_address" value="[fd00:1122:3344:101::3]:32221"/> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::3]:32222"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7042024-07-19T17:45:01.830ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:465 zone = oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a
7052024-07-19T17:45:01.836ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_15be01e5-1765-45f0-b521-da99b395bdb3: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService9"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::5"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="[fd00:1122:3344:101::5]:32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value="15be01e5-1765-45f0-b521-da99b395bdb3"/> <propval type="astring" name="cockroach_address" value="[fd00:1122:3344:101::5]:32221"/> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::5]:32222"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7062024-07-19T17:45:01.837ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService5"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::6"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="[fd00:1122:3344:101::6]:32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value="ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0"/> <propval type="astring" name="cockroach_address" value="[fd00:1122:3344:101::6]:32221"/> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::6]:32222"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7072024-07-19T17:45:01.837ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:465 zone = oxz_cockroachdb_15be01e5-1765-45f0-b521-da99b395bdb3
7082024-07-19T17:45:01.842ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:465 zone = oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0
7092024-07-19T17:45:01.960ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c168b27b-f467-4350-a348-4006ce85a213") failed: Property not found. retry in 63.930396ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c168b27b-f467-4350-a348-4006ce85a213
7102024-07-19T17:45:01.968ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d8a1310d-09c1-4844-91b7-509f93b30cfd") failed: Property not found. retry in 52.612946ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d8a1310d-09c1-4844-91b7-509f93b30cfd
7112024-07-19T17:45:01.976ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a") failed: Property not found. retry in 35.213877ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a
7122024-07-19T17:45:01.983ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_15be01e5-1765-45f0-b521-da99b395bdb3") failed: Property not found. retry in 35.034651ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_15be01e5-1765-45f0-b521-da99b395bdb3
7132024-07-19T17:45:01.989ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0") failed: Property not found. retry in 56.745125ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0
7142024-07-19T17:45:02.018ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a") failed: Property not found. retry in 96.743925ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a
7152024-07-19T17:45:02.037ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_15be01e5-1765-45f0-b521-da99b395bdb3") failed: Property not found. retry in 84.50005ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_15be01e5-1765-45f0-b521-da99b395bdb3
7162024-07-19T17:45:02.214ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d8a1310d-09c1-4844-91b7-509f93b30cfd") failed: Property not found. retry in 76.390393ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d8a1310d-09c1-4844-91b7-509f93b30cfd
7172024-07-19T17:45:02.295ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a") failed: Property not found. retry in 208.272695ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a
7182024-07-19T17:45:02.457ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c168b27b-f467-4350-a348-4006ce85a213") failed: Property not found. retry in 109.733936ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c168b27b-f467-4350-a348-4006ce85a213
7192024-07-19T17:45:02.606ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_15be01e5-1765-45f0-b521-da99b395bdb3") failed: Property not found. retry in 271.428436ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_15be01e5-1765-45f0-b521-da99b395bdb3
7202024-07-19T17:45:02.806ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0") failed: Property not found. retry in 142.397986ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0
7212024-07-19T17:45:02.996ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d8a1310d-09c1-4844-91b7-509f93b30cfd") failed: Property not found. retry in 147.50577ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d8a1310d-09c1-4844-91b7-509f93b30cfd
7222024-07-19T17:45:03.216ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a") failed: Property not found. retry in 489.167177ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a
7232024-07-19T17:45:03.428ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c168b27b-f467-4350-a348-4006ce85a213") failed: Property not found. retry in 111.338854ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c168b27b-f467-4350-a348-4006ce85a213
7242024-07-19T17:45:03.787ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_15be01e5-1765-45f0-b521-da99b395bdb3") failed: Property not found. retry in 450.037456ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_15be01e5-1765-45f0-b521-da99b395bdb3
7252024-07-19T17:45:04.934ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0") failed: Property not found. retry in 220.269867ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0
7262024-07-19T17:45:04.954ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d8a1310d-09c1-4844-91b7-509f93b30cfd") failed: Property not found. retry in 431.367451ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d8a1310d-09c1-4844-91b7-509f93b30cfd
7272024-07-19T17:45:04.971ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a") failed: Property not found. retry in 418.393264ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a
7282024-07-19T17:45:04.980ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c168b27b-f467-4350-a348-4006ce85a213") failed: Property not found. retry in 458.592722ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c168b27b-f467-4350-a348-4006ce85a213
7292024-07-19T17:45:04.991ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_15be01e5-1765-45f0-b521-da99b395bdb3") failed: Property not found. retry in 787.323773ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_15be01e5-1765-45f0-b521-da99b395bdb3
7302024-07-19T17:45:05.164ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0") failed: Property not found. retry in 532.530582ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0
7312024-07-19T17:45:05.397ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d8a1310d-09c1-4844-91b7-509f93b30cfd") failed: Property not found. retry in 465.74801ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d8a1310d-09c1-4844-91b7-509f93b30cfd
7322024-07-19T17:45:05.407ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a") failed: Property not found. retry in 533.012018ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a
7332024-07-19T17:45:05.451ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c168b27b-f467-4350-a348-4006ce85a213") failed: Property not found. retry in 715.095706ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c168b27b-f467-4350-a348-4006ce85a213
7342024-07-19T17:45:05.708ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0") failed: Property not found. retry in 977.976353ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0
7352024-07-19T17:45:05.789ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_15be01e5-1765-45f0-b521-da99b395bdb3") failed: Property not found. retry in 1.275006675s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_15be01e5-1765-45f0-b521-da99b395bdb3
7362024-07-19T17:45:05.873ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d8a1310d-09c1-4844-91b7-509f93b30cfd") failed: Property not found. retry in 534.993388ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d8a1310d-09c1-4844-91b7-509f93b30cfd
7372024-07-19T17:45:05.950ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a") failed: Property not found. retry in 780.493332ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a
7382024-07-19T17:45:06.177ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c168b27b-f467-4350-a348-4006ce85a213") failed: Property not found. retry in 1.186094682s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c168b27b-f467-4350-a348-4006ce85a213
7392024-07-19T17:45:06.418ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d8a1310d-09c1-4844-91b7-509f93b30cfd") failed: Property not found. retry in 1.296585222s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d8a1310d-09c1-4844-91b7-509f93b30cfd
7402024-07-19T17:45:06.695ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0") failed: Property not found. retry in 815.863224ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0
7412024-07-19T17:45:06.741ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a") failed: Property not found. retry in 1.148301121s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a
7422024-07-19T17:45:07.074ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_15be01e5-1765-45f0-b521-da99b395bdb3") failed: Property not found. retry in 1.493401336s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_15be01e5-1765-45f0-b521-da99b395bdb3
7432024-07-19T17:45:07.374ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c168b27b-f467-4350-a348-4006ce85a213") failed: Property not found. retry in 833.469833ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c168b27b-f467-4350-a348-4006ce85a213
7442024-07-19T17:45:07.521ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0") failed: Property not found. retry in 658.426474ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0
7452024-07-19T17:45:07.726ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d8a1310d-09c1-4844-91b7-509f93b30cfd") failed: Property not found. retry in 1.429889758s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d8a1310d-09c1-4844-91b7-509f93b30cfd
7462024-07-19T17:45:07.904ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a") failed: Property not found. retry in 911.430388ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a
7472024-07-19T17:45:08.189ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0") failed: Property not found. retry in 1.199340696s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0
7482024-07-19T17:45:08.219ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c168b27b-f467-4350-a348-4006ce85a213") failed: Property not found. retry in 1.123499225s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c168b27b-f467-4350-a348-4006ce85a213
7492024-07-19T17:45:08.577ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_15be01e5-1765-45f0-b521-da99b395bdb3") failed: Property not found. retry in 649.209682ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_15be01e5-1765-45f0-b521-da99b395bdb3
7502024-07-19T17:45:08.825ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a") failed: Property not found. retry in 1.470089415s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a
7512024-07-19T17:45:09.164ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d8a1310d-09c1-4844-91b7-509f93b30cfd") failed: Property not found. retry in 690.997896ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d8a1310d-09c1-4844-91b7-509f93b30cfd
7522024-07-19T17:45:09.234ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_15be01e5-1765-45f0-b521-da99b395bdb3") failed: Property not found. retry in 711.730983ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_15be01e5-1765-45f0-b521-da99b395bdb3
7532024-07-19T17:45:09.350ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c168b27b-f467-4350-a348-4006ce85a213") failed: Property not found. retry in 812.955809ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c168b27b-f467-4350-a348-4006ce85a213
7542024-07-19T17:45:09.397ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0") failed: Property not found. retry in 1.494394545s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0
7552024-07-19T17:45:09.866ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d8a1310d-09c1-4844-91b7-509f93b30cfd") failed: Property not found. retry in 586.117533ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d8a1310d-09c1-4844-91b7-509f93b30cfd
7562024-07-19T17:45:09.956ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_15be01e5-1765-45f0-b521-da99b395bdb3") failed: Property not found. retry in 1.141115546s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_15be01e5-1765-45f0-b521-da99b395bdb3
7572024-07-19T17:45:10.173ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c168b27b-f467-4350-a348-4006ce85a213") failed: Property not found. retry in 651.955617ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c168b27b-f467-4350-a348-4006ce85a213
7582024-07-19T17:45:10.304ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a") failed: Property not found. retry in 1.400502517s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a
7592024-07-19T17:45:10.462ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d8a1310d-09c1-4844-91b7-509f93b30cfd") failed: Property not found. retry in 601.53495ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d8a1310d-09c1-4844-91b7-509f93b30cfd
7602024-07-19T17:45:10.836ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c168b27b-f467-4350-a348-4006ce85a213") failed: Property not found. retry in 761.751123ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c168b27b-f467-4350-a348-4006ce85a213
7612024-07-19T17:45:10.902ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0") failed: Property not found. retry in 1.357184568s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0
7622024-07-19T17:45:11.075ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d8a1310d-09c1-4844-91b7-509f93b30cfd") failed: Property not found. retry in 929.146512ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d8a1310d-09c1-4844-91b7-509f93b30cfd
7632024-07-19T17:45:11.106ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_15be01e5-1765-45f0-b521-da99b395bdb3") failed: Property not found. retry in 1.320888348s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_15be01e5-1765-45f0-b521-da99b395bdb3
7642024-07-19T17:45:11.326ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 309.766481ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
7652024-07-19T17:45:11.716ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a") failed: Property not found. retry in 1.304291706s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a
7662024-07-19T17:45:12.270ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0") failed: Property not found. retry in 635.253597ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0
7672024-07-19T17:45:12.916ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0") failed: Property not found. retry in 837.767807ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0
7682024-07-19T17:45:13.031ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a") failed: Property not found. retry in 1.088558554s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a
7692024-07-19T17:45:13.939ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0") failed: Property not found. retry in 863.197083ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0
7702024-07-19T17:45:13.972ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:48585
7712024-07-19T17:45:13.972ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 119 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:48585 req_id = daed91d8-34be-42a7-918e-ee590d137ac0 response_code = 200 uri = /baseboard
7722024-07-19T17:45:14.133ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a") failed: Property not found. retry in 1.43789946s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a
7732024-07-19T17:45:14.814ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0") failed: Property not found. retry in 1.344516934s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0
7742024-07-19T17:45:15.582ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a") failed: Property not found. retry in 1.384384924s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a
7752024-07-19T17:45:16.168ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0") failed: Property not found. retry in 1.175372149s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0
7762024-07-19T17:45:16.977ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a") failed: Property not found. retry in 1.048625767s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a
7772024-07-19T17:45:17.353ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0") failed: Property not found. retry in 1.433989244s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0
7782024-07-19T17:45:18.036ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a") failed: Property not found. retry in 801.913294ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a
7792024-07-19T17:45:18.886ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0") failed: Property not found. retry in 1.37179846s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0
7802024-07-19T17:45:19.034ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a") failed: Property not found. retry in 1.339428903s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a
7812024-07-19T17:45:20.265ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0") failed: Property not found. retry in 558.136282ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0
7822024-07-19T17:45:20.382ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a") failed: Property not found. retry in 1.490319513s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a
7832024-07-19T17:45:20.832ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0") failed: Property not found. retry in 1.434698707s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0
7842024-07-19T17:45:22.288ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2980 zone = oxz_cockroachdb_15be01e5-1765-45f0-b521-da99b395bdb3
7852024-07-19T17:45:22.289ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2980 zone = oxz_cockroachdb_ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0
7862024-07-19T17:45:22.289ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2980 zone = oxz_cockroachdb_d8a1310d-09c1-4844-91b7-509f93b30cfd
7872024-07-19T17:45:22.289ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2980 zone = oxz_cockroachdb_27c74baf-8069-4296-ac26-2f4818d2ab3a
7882024-07-19T17:45:22.289ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2980 zone = oxz_cockroachdb_c168b27b-f467-4350-a348-4006ce85a213
7892024-07-19T17:45:22.289ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
7902024-07-19T17:45:22.289ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
7912024-07-19T17:45:22.290ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 24041550 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:54642 req_id = d218f235-00fb-4a12-949a-28430e9b3d5b response_code = 204 uri = /omicron-zones
7922024-07-19T17:45:22.324ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:38809
7932024-07-19T17:45:22.351ZINFOSledAgent (ServiceManager): Initializing CRDB Cluster - sending request to [fd00:1122:3344:101::5]:32221 file = sled-agent/src/services.rs:3403
7942024-07-19T17:45:22.709ZINFOSledAgent (ServiceManager): Formatting CRDB file = sled-agent/src/services.rs:3421
7952024-07-19T17:45:26.673ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 994.43046ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
7962024-07-19T17:45:42.703ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 1.203622472s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
7972024-07-19T17:45:44.008ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:55204
7982024-07-19T17:45:44.008ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 40 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:55204 req_id = 814b755f-b4ab-4b80-807b-7a505171eeb5 response_code = 200 uri = /baseboard
7992024-07-19T17:45:44.740ZINFOSledAgent (ServiceManager): Formatting CRDB - Completed file = sled-agent/src/services.rs:3444
8002024-07-19T17:45:44.740ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 22416303 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:38809 req_id = 364d84ca-532d-4e5d-aca9-4a5120bef68c response_code = 204 uri = /cockroachdb
8012024-07-19T17:45:44.772ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 8ae14f97-bd01-42ad-87d5-23eae9194521, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: bc60810c-eece-4cbd-a0f8-d04271924b7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bf3fc707-b452-4d30-a92b-e56b4d4122ee, kind: Service { id: bc60810c-eece-4cbd-a0f8-d04271924b7c }, name: Name("ntp-bc60810c-eece-4cbd-a0f8-d04271924b7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 196])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: c168b27b-f467-4350-a348-4006ce85a213, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 15be01e5-1765-45f0-b521-da99b395bdb3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: d8a1310d-09c1-4844-91b7-509f93b30cfd, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 59731bd7-ffc9-493f-a33c-a1cf5ba2df30, kind: Service { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1 }, name: Name("external-dns-1eb90f9c-ba99-428e-885b-5fa2df7ae2f1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 32])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 09fdc6fd-8297-436e-903d-b18710599203, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 005b2740-84d5-473e-ae59-aa79fd2fe5b6, kind: Service { id: 09fdc6fd-8297-436e-903d-b18710599203 }, name: Name("external-dns-09fdc6fd-8297-436e-903d-b18710599203"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 6])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: c6edf68c-f558-4f51-9aed-6eb91fb3ad51, kind: Service { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52 }, name: Name("nexus-87d65a87-e1e5-43e4-9cba-b4afd5e9ac52"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 117])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6088410e-a8fc-4d33-b593-6385e863291f, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: e6898038-bf11-41c8-b971-6ff3eed6d38b, kind: Service { id: 6088410e-a8fc-4d33-b593-6385e863291f }, name: Name("nexus-6088410e-a8fc-4d33-b593-6385e863291f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 27])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 742e0330-77de-4723-a145-74751d41c73b, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: ffe6d41d-1349-42b2-a887-49d734eef422, kind: Service { id: 742e0330-77de-4723-a145-74751d41c73b }, name: Name("nexus-742e0330-77de-4723-a145-74751d41c73b"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 199, 184])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8d644923-5967-4351-a1ad-fe8a938f33e1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 740bd3e0-b30c-4033-968a-5451c8d5adcd, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 3286a4ca-fffd-4ed0-9c21-d528e1d23603, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 0b1925c8-3ae5-4372-a3de-213551574fe1, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 663bd1f7-7a29-484c-bfce-d312646b1660, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: c6d325c3-cddf-4528-8ba6-a083f6511284, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External } } } }, OmicronZoneConfig { id: c30b7bc1-fd10-4942-8055-ada48d2e3957, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External } } } }, OmicronZoneConfig { id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
8022024-07-19T17:45:44.772ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:37752
8032024-07-19T17:45:44.772ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
8042024-07-19T17:45:44.772ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8052024-07-19T17:45:44.798ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
8062024-07-19T17:45:44.798ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }
8072024-07-19T17:45:44.820ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
8082024-07-19T17:45:44.820ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8092024-07-19T17:45:44.839ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
8102024-07-19T17:45:44.839ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8112024-07-19T17:45:44.859ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
8122024-07-19T17:45:44.859ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8132024-07-19T17:45:44.879ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
8142024-07-19T17:45:44.879ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8152024-07-19T17:45:44.899ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
8162024-07-19T17:45:44.899ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8172024-07-19T17:45:44.919ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
8182024-07-19T17:45:44.919ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8192024-07-19T17:45:44.939ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
8202024-07-19T17:45:44.939ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
8212024-07-19T17:45:45.028ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
8222024-07-19T17:45:45.028ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
8232024-07-19T17:45:45.109ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
8242024-07-19T17:45:45.109ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
8252024-07-19T17:45:45.197ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
8262024-07-19T17:45:45.197ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }
8272024-07-19T17:45:45.286ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
8282024-07-19T17:45:45.286ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }
8292024-07-19T17:45:45.358ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
8302024-07-19T17:45:45.358ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }
8312024-07-19T17:45:45.446ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
8322024-07-19T17:45:45.446ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }
8332024-07-19T17:45:45.527ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
8342024-07-19T17:45:45.527ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }
8352024-07-19T17:45:45.599ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
8362024-07-19T17:45:45.599ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }
8372024-07-19T17:45:45.672ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
8382024-07-19T17:45:45.672ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }
8392024-07-19T17:45:45.750ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
8402024-07-19T17:45:45.750ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }
8412024-07-19T17:45:45.831ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
8422024-07-19T17:45:45.831ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }
8432024-07-19T17:45:45.908ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
8442024-07-19T17:45:45.908ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/omicron-zones.json file = common/src/ledger.rs:177
8452024-07-19T17:45:45.908ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/omicron-zones.json file = common/src/ledger.rs:177
8462024-07-19T17:45:45.909ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
8472024-07-19T17:45:46.015ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
8482024-07-19T17:45:46.015ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
8492024-07-19T17:45:46.015ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
8502024-07-19T17:45:46.015ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
8512024-07-19T17:45:46.015ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
8522024-07-19T17:45:46.015ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
8532024-07-19T17:45:46.015ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
8542024-07-19T17:45:46.015ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
8552024-07-19T17:45:46.015ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
8562024-07-19T17:45:46.015ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
8572024-07-19T17:45:46.015ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
8582024-07-19T17:45:46.015ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
8592024-07-19T17:45:46.015ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
8602024-07-19T17:45:46.015ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
8612024-07-19T17:45:46.015ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
8622024-07-19T17:45:46.015ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
8632024-07-19T17:45:46.015ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
8642024-07-19T17:45:46.015ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
8652024-07-19T17:45:46.015ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
8662024-07-19T17:45:46.015ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b", TimeNotSynchronized), ("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17", TimeNotSynchronized), ("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52", TimeNotSynchronized), ("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203", TimeNotSynchronized), ("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1", TimeNotSynchronized), ("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1", TimeNotSynchronized), ("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1", TimeNotSynchronized), ("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9", TimeNotSynchronized), ("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957", TimeNotSynchronized), ("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7", TimeNotSynchronized), ("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603", TimeNotSynchronized), ("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f", TimeNotSynchronized), ("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec", TimeNotSynchronized), ("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1", TimeNotSynchronized), ("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd", TimeNotSynchronized), ("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660", TimeNotSynchronized), ("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb", TimeNotSynchronized), ("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b", TimeNotSynchronized), ("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 1242742 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:37752 req_id = 947ca388-168b-44e3-ae08-ffc91da4dace response_code = 500 uri = /omicron-zones
8672024-07-19T17:45:46.015ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "947ca388-168b-44e3-ae08-ffc91da4dace", "content-length": "124", "date": "Fri, 19 Jul 2024 17:45:46 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "947ca388-168b-44e3-ae08-ffc91da4dace" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 101.356559ms sled_address = [fd00:1122:3344:101::1]:12345
8682024-07-19T17:45:46.117ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 8ae14f97-bd01-42ad-87d5-23eae9194521, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: bc60810c-eece-4cbd-a0f8-d04271924b7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bf3fc707-b452-4d30-a92b-e56b4d4122ee, kind: Service { id: bc60810c-eece-4cbd-a0f8-d04271924b7c }, name: Name("ntp-bc60810c-eece-4cbd-a0f8-d04271924b7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 196])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: c168b27b-f467-4350-a348-4006ce85a213, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 15be01e5-1765-45f0-b521-da99b395bdb3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: d8a1310d-09c1-4844-91b7-509f93b30cfd, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 59731bd7-ffc9-493f-a33c-a1cf5ba2df30, kind: Service { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1 }, name: Name("external-dns-1eb90f9c-ba99-428e-885b-5fa2df7ae2f1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 32])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 09fdc6fd-8297-436e-903d-b18710599203, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 005b2740-84d5-473e-ae59-aa79fd2fe5b6, kind: Service { id: 09fdc6fd-8297-436e-903d-b18710599203 }, name: Name("external-dns-09fdc6fd-8297-436e-903d-b18710599203"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 6])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: c6edf68c-f558-4f51-9aed-6eb91fb3ad51, kind: Service { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52 }, name: Name("nexus-87d65a87-e1e5-43e4-9cba-b4afd5e9ac52"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 117])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6088410e-a8fc-4d33-b593-6385e863291f, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: e6898038-bf11-41c8-b971-6ff3eed6d38b, kind: Service { id: 6088410e-a8fc-4d33-b593-6385e863291f }, name: Name("nexus-6088410e-a8fc-4d33-b593-6385e863291f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 27])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 742e0330-77de-4723-a145-74751d41c73b, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: ffe6d41d-1349-42b2-a887-49d734eef422, kind: Service { id: 742e0330-77de-4723-a145-74751d41c73b }, name: Name("nexus-742e0330-77de-4723-a145-74751d41c73b"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 199, 184])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8d644923-5967-4351-a1ad-fe8a938f33e1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 740bd3e0-b30c-4033-968a-5451c8d5adcd, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 3286a4ca-fffd-4ed0-9c21-d528e1d23603, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 0b1925c8-3ae5-4372-a3de-213551574fe1, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 663bd1f7-7a29-484c-bfce-d312646b1660, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: c6d325c3-cddf-4528-8ba6-a083f6511284, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External } } } }, OmicronZoneConfig { id: c30b7bc1-fd10-4942-8055-ada48d2e3957, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External } } } }, OmicronZoneConfig { id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
8692024-07-19T17:45:46.118ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
8702024-07-19T17:45:46.118ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8712024-07-19T17:45:46.141ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
8722024-07-19T17:45:46.141ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }
8732024-07-19T17:45:46.162ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
8742024-07-19T17:45:46.162ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8752024-07-19T17:45:46.179ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
8762024-07-19T17:45:46.179ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8772024-07-19T17:45:46.197ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
8782024-07-19T17:45:46.197ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8792024-07-19T17:45:46.214ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
8802024-07-19T17:45:46.214ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8812024-07-19T17:45:46.233ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
8822024-07-19T17:45:46.233ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8832024-07-19T17:45:46.251ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
8842024-07-19T17:45:46.251ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8852024-07-19T17:45:46.270ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
8862024-07-19T17:45:46.270ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
8872024-07-19T17:45:46.292ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
8882024-07-19T17:45:46.292ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
8892024-07-19T17:45:46.312ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
8902024-07-19T17:45:46.312ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
8912024-07-19T17:45:46.333ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
8922024-07-19T17:45:46.333ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }
8932024-07-19T17:45:46.353ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
8942024-07-19T17:45:46.353ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }
8952024-07-19T17:45:46.371ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
8962024-07-19T17:45:46.371ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }
8972024-07-19T17:45:46.392ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
8982024-07-19T17:45:46.392ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }
8992024-07-19T17:45:46.409ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
9002024-07-19T17:45:46.409ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }
9012024-07-19T17:45:46.426ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
9022024-07-19T17:45:46.426ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }
9032024-07-19T17:45:46.442ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
9042024-07-19T17:45:46.442ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }
9052024-07-19T17:45:46.460ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
9062024-07-19T17:45:46.460ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }
9072024-07-19T17:45:46.477ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
9082024-07-19T17:45:46.477ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }
9092024-07-19T17:45:46.494ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
9102024-07-19T17:45:46.494ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/omicron-zones.json file = common/src/ledger.rs:177
9112024-07-19T17:45:46.495ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/omicron-zones.json file = common/src/ledger.rs:177
9122024-07-19T17:45:46.495ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
9132024-07-19T17:45:46.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
9142024-07-19T17:45:46.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
9152024-07-19T17:45:46.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
9162024-07-19T17:45:46.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
9172024-07-19T17:45:46.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
9182024-07-19T17:45:46.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
9192024-07-19T17:45:46.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
9202024-07-19T17:45:46.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
9212024-07-19T17:45:46.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
9222024-07-19T17:45:46.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
9232024-07-19T17:45:46.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
9242024-07-19T17:45:46.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
9252024-07-19T17:45:46.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
9262024-07-19T17:45:46.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
9272024-07-19T17:45:46.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
9282024-07-19T17:45:46.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
9292024-07-19T17:45:46.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
9302024-07-19T17:45:46.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
9312024-07-19T17:45:46.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
9322024-07-19T17:45:46.589ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1", TimeNotSynchronized), ("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957", TimeNotSynchronized), ("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17", TimeNotSynchronized), ("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb", TimeNotSynchronized), ("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660", TimeNotSynchronized), ("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203", TimeNotSynchronized), ("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd", TimeNotSynchronized), ("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9", TimeNotSynchronized), ("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52", TimeNotSynchronized), ("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284", TimeNotSynchronized), ("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b", TimeNotSynchronized), ("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603", TimeNotSynchronized), ("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7", TimeNotSynchronized), ("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1", TimeNotSynchronized), ("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1", TimeNotSynchronized), ("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f", TimeNotSynchronized), ("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b", TimeNotSynchronized), ("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec", TimeNotSynchronized), ("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 471622 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:37752 req_id = 6381de54-29df-4b3c-87e0-ce76aecdb3cb response_code = 500 uri = /omicron-zones
9332024-07-19T17:45:46.590ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "6381de54-29df-4b3c-87e0-ce76aecdb3cb", "content-length": "124", "date": "Fri, 19 Jul 2024 17:45:46 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "6381de54-29df-4b3c-87e0-ce76aecdb3cb" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 152.690002ms sled_address = [fd00:1122:3344:101::1]:12345
9342024-07-19T17:45:46.743ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 8ae14f97-bd01-42ad-87d5-23eae9194521, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: bc60810c-eece-4cbd-a0f8-d04271924b7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bf3fc707-b452-4d30-a92b-e56b4d4122ee, kind: Service { id: bc60810c-eece-4cbd-a0f8-d04271924b7c }, name: Name("ntp-bc60810c-eece-4cbd-a0f8-d04271924b7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 196])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: c168b27b-f467-4350-a348-4006ce85a213, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 15be01e5-1765-45f0-b521-da99b395bdb3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: d8a1310d-09c1-4844-91b7-509f93b30cfd, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 59731bd7-ffc9-493f-a33c-a1cf5ba2df30, kind: Service { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1 }, name: Name("external-dns-1eb90f9c-ba99-428e-885b-5fa2df7ae2f1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 32])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 09fdc6fd-8297-436e-903d-b18710599203, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 005b2740-84d5-473e-ae59-aa79fd2fe5b6, kind: Service { id: 09fdc6fd-8297-436e-903d-b18710599203 }, name: Name("external-dns-09fdc6fd-8297-436e-903d-b18710599203"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 6])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: c6edf68c-f558-4f51-9aed-6eb91fb3ad51, kind: Service { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52 }, name: Name("nexus-87d65a87-e1e5-43e4-9cba-b4afd5e9ac52"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 117])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6088410e-a8fc-4d33-b593-6385e863291f, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: e6898038-bf11-41c8-b971-6ff3eed6d38b, kind: Service { id: 6088410e-a8fc-4d33-b593-6385e863291f }, name: Name("nexus-6088410e-a8fc-4d33-b593-6385e863291f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 27])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 742e0330-77de-4723-a145-74751d41c73b, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: ffe6d41d-1349-42b2-a887-49d734eef422, kind: Service { id: 742e0330-77de-4723-a145-74751d41c73b }, name: Name("nexus-742e0330-77de-4723-a145-74751d41c73b"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 199, 184])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8d644923-5967-4351-a1ad-fe8a938f33e1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 740bd3e0-b30c-4033-968a-5451c8d5adcd, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 3286a4ca-fffd-4ed0-9c21-d528e1d23603, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 0b1925c8-3ae5-4372-a3de-213551574fe1, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 663bd1f7-7a29-484c-bfce-d312646b1660, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: c6d325c3-cddf-4528-8ba6-a083f6511284, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External } } } }, OmicronZoneConfig { id: c30b7bc1-fd10-4942-8055-ada48d2e3957, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External } } } }, OmicronZoneConfig { id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
9352024-07-19T17:45:46.743ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
9362024-07-19T17:45:46.743ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9372024-07-19T17:45:46.768ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
9382024-07-19T17:45:46.768ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }
9392024-07-19T17:45:46.789ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
9402024-07-19T17:45:46.789ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9412024-07-19T17:45:46.811ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
9422024-07-19T17:45:46.812ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9432024-07-19T17:45:46.832ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
9442024-07-19T17:45:46.832ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9452024-07-19T17:45:46.852ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
9462024-07-19T17:45:46.852ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9472024-07-19T17:45:46.872ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
9482024-07-19T17:45:46.872ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9492024-07-19T17:45:46.893ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
9502024-07-19T17:45:46.893ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9512024-07-19T17:45:46.914ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
9522024-07-19T17:45:46.914ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
9532024-07-19T17:45:46.939ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
9542024-07-19T17:45:46.940ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
9552024-07-19T17:45:46.962ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
9562024-07-19T17:45:46.962ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
9572024-07-19T17:45:46.983ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
9582024-07-19T17:45:46.983ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }
9592024-07-19T17:45:47.002ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
9602024-07-19T17:45:47.002ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }
9612024-07-19T17:45:47.024ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
9622024-07-19T17:45:47.024ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }
9632024-07-19T17:45:47.046ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
9642024-07-19T17:45:47.046ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }
9652024-07-19T17:45:47.068ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
9662024-07-19T17:45:47.068ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }
9672024-07-19T17:45:47.089ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
9682024-07-19T17:45:47.090ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }
9692024-07-19T17:45:47.110ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
9702024-07-19T17:45:47.110ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }
9712024-07-19T17:45:47.132ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
9722024-07-19T17:45:47.132ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }
9732024-07-19T17:45:47.153ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
9742024-07-19T17:45:47.153ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }
9752024-07-19T17:45:47.174ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
9762024-07-19T17:45:47.174ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/omicron-zones.json file = common/src/ledger.rs:177
9772024-07-19T17:45:47.175ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/omicron-zones.json file = common/src/ledger.rs:177
9782024-07-19T17:45:47.175ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
9792024-07-19T17:45:47.273ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
9802024-07-19T17:45:47.273ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
9812024-07-19T17:45:47.273ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
9822024-07-19T17:45:47.273ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
9832024-07-19T17:45:47.273ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
9842024-07-19T17:45:47.273ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
9852024-07-19T17:45:47.274ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
9862024-07-19T17:45:47.274ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
9872024-07-19T17:45:47.274ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
9882024-07-19T17:45:47.274ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
9892024-07-19T17:45:47.274ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
9902024-07-19T17:45:47.274ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
9912024-07-19T17:45:47.274ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
9922024-07-19T17:45:47.274ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
9932024-07-19T17:45:47.274ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
9942024-07-19T17:45:47.274ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
9952024-07-19T17:45:47.274ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
9962024-07-19T17:45:47.274ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
9972024-07-19T17:45:47.274ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
9982024-07-19T17:45:47.274ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b", TimeNotSynchronized), ("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec", TimeNotSynchronized), ("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7", TimeNotSynchronized), ("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9", TimeNotSynchronized), ("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1", TimeNotSynchronized), ("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203", TimeNotSynchronized), ("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1", TimeNotSynchronized), ("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b", TimeNotSynchronized), ("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1", TimeNotSynchronized), ("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660", TimeNotSynchronized), ("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f", TimeNotSynchronized), ("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52", TimeNotSynchronized), ("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603", TimeNotSynchronized), ("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1", TimeNotSynchronized), ("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284", TimeNotSynchronized), ("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb", TimeNotSynchronized), ("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd", TimeNotSynchronized), ("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17", TimeNotSynchronized), ("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 530769 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:37752 req_id = 195d9923-8011-4b0a-8dcb-60959b54ea3a response_code = 500 uri = /omicron-zones
9992024-07-19T17:45:47.274ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "195d9923-8011-4b0a-8dcb-60959b54ea3a", "content-length": "124", "date": "Fri, 19 Jul 2024 17:45:47 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "195d9923-8011-4b0a-8dcb-60959b54ea3a" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 188.139096ms sled_address = [fd00:1122:3344:101::1]:12345
10002024-07-19T17:45:47.463ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 8ae14f97-bd01-42ad-87d5-23eae9194521, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: bc60810c-eece-4cbd-a0f8-d04271924b7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bf3fc707-b452-4d30-a92b-e56b4d4122ee, kind: Service { id: bc60810c-eece-4cbd-a0f8-d04271924b7c }, name: Name("ntp-bc60810c-eece-4cbd-a0f8-d04271924b7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 196])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: c168b27b-f467-4350-a348-4006ce85a213, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 15be01e5-1765-45f0-b521-da99b395bdb3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: d8a1310d-09c1-4844-91b7-509f93b30cfd, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 59731bd7-ffc9-493f-a33c-a1cf5ba2df30, kind: Service { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1 }, name: Name("external-dns-1eb90f9c-ba99-428e-885b-5fa2df7ae2f1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 32])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 09fdc6fd-8297-436e-903d-b18710599203, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 005b2740-84d5-473e-ae59-aa79fd2fe5b6, kind: Service { id: 09fdc6fd-8297-436e-903d-b18710599203 }, name: Name("external-dns-09fdc6fd-8297-436e-903d-b18710599203"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 6])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: c6edf68c-f558-4f51-9aed-6eb91fb3ad51, kind: Service { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52 }, name: Name("nexus-87d65a87-e1e5-43e4-9cba-b4afd5e9ac52"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 117])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6088410e-a8fc-4d33-b593-6385e863291f, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: e6898038-bf11-41c8-b971-6ff3eed6d38b, kind: Service { id: 6088410e-a8fc-4d33-b593-6385e863291f }, name: Name("nexus-6088410e-a8fc-4d33-b593-6385e863291f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 27])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 742e0330-77de-4723-a145-74751d41c73b, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: ffe6d41d-1349-42b2-a887-49d734eef422, kind: Service { id: 742e0330-77de-4723-a145-74751d41c73b }, name: Name("nexus-742e0330-77de-4723-a145-74751d41c73b"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 199, 184])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8d644923-5967-4351-a1ad-fe8a938f33e1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 740bd3e0-b30c-4033-968a-5451c8d5adcd, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 3286a4ca-fffd-4ed0-9c21-d528e1d23603, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 0b1925c8-3ae5-4372-a3de-213551574fe1, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 663bd1f7-7a29-484c-bfce-d312646b1660, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: c6d325c3-cddf-4528-8ba6-a083f6511284, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External } } } }, OmicronZoneConfig { id: c30b7bc1-fd10-4942-8055-ada48d2e3957, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External } } } }, OmicronZoneConfig { id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
10012024-07-19T17:45:47.464ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
10022024-07-19T17:45:47.464ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10032024-07-19T17:45:47.488ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
10042024-07-19T17:45:47.488ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }
10052024-07-19T17:45:47.509ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
10062024-07-19T17:45:47.509ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10072024-07-19T17:45:47.529ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
10082024-07-19T17:45:47.529ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10092024-07-19T17:45:47.549ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
10102024-07-19T17:45:47.549ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10112024-07-19T17:45:47.569ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
10122024-07-19T17:45:47.569ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10132024-07-19T17:45:47.589ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
10142024-07-19T17:45:47.589ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10152024-07-19T17:45:47.608ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
10162024-07-19T17:45:47.608ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10172024-07-19T17:45:47.628ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
10182024-07-19T17:45:47.629ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
10192024-07-19T17:45:47.648ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
10202024-07-19T17:45:47.648ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
10212024-07-19T17:45:47.668ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
10222024-07-19T17:45:47.668ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
10232024-07-19T17:45:47.688ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
10242024-07-19T17:45:47.688ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }
10252024-07-19T17:45:47.708ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
10262024-07-19T17:45:47.708ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }
10272024-07-19T17:45:47.728ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
10282024-07-19T17:45:47.728ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }
10292024-07-19T17:45:47.747ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
10302024-07-19T17:45:47.747ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }
10312024-07-19T17:45:47.767ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
10322024-07-19T17:45:47.767ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }
10332024-07-19T17:45:47.786ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
10342024-07-19T17:45:47.786ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }
10352024-07-19T17:45:47.807ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
10362024-07-19T17:45:47.807ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }
10372024-07-19T17:45:47.828ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
10382024-07-19T17:45:47.828ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }
10392024-07-19T17:45:47.848ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
10402024-07-19T17:45:47.848ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }
10412024-07-19T17:45:47.869ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
10422024-07-19T17:45:47.869ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/omicron-zones.json file = common/src/ledger.rs:177
10432024-07-19T17:45:47.869ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/omicron-zones.json file = common/src/ledger.rs:177
10442024-07-19T17:45:47.869ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
10452024-07-19T17:45:47.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
10462024-07-19T17:45:47.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
10472024-07-19T17:45:47.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
10482024-07-19T17:45:47.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
10492024-07-19T17:45:47.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
10502024-07-19T17:45:47.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
10512024-07-19T17:45:47.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
10522024-07-19T17:45:47.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
10532024-07-19T17:45:47.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
10542024-07-19T17:45:47.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
10552024-07-19T17:45:47.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
10562024-07-19T17:45:47.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
10572024-07-19T17:45:47.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
10582024-07-19T17:45:47.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
10592024-07-19T17:45:47.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
10602024-07-19T17:45:47.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
10612024-07-19T17:45:47.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
10622024-07-19T17:45:47.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
10632024-07-19T17:45:47.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
10642024-07-19T17:45:47.964ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1", TimeNotSynchronized), ("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec", TimeNotSynchronized), ("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b", TimeNotSynchronized), ("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1", TimeNotSynchronized), ("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd", TimeNotSynchronized), ("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1", TimeNotSynchronized), ("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17", TimeNotSynchronized), ("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203", TimeNotSynchronized), ("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284", TimeNotSynchronized), ("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b", TimeNotSynchronized), ("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb", TimeNotSynchronized), ("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52", TimeNotSynchronized), ("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660", TimeNotSynchronized), ("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603", TimeNotSynchronized), ("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1", TimeNotSynchronized), ("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9", TimeNotSynchronized), ("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7", TimeNotSynchronized), ("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957", TimeNotSynchronized), ("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 501034 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:37752 req_id = bdaca22f-e672-4895-8223-0883f4c48399 response_code = 500 uri = /omicron-zones
10652024-07-19T17:45:47.965ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "bdaca22f-e672-4895-8223-0883f4c48399", "content-length": "124", "date": "Fri, 19 Jul 2024 17:45:47 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "bdaca22f-e672-4895-8223-0883f4c48399" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 163.372244ms sled_address = [fd00:1122:3344:101::1]:12345
10662024-07-19T17:45:48.129ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 8ae14f97-bd01-42ad-87d5-23eae9194521, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: bc60810c-eece-4cbd-a0f8-d04271924b7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bf3fc707-b452-4d30-a92b-e56b4d4122ee, kind: Service { id: bc60810c-eece-4cbd-a0f8-d04271924b7c }, name: Name("ntp-bc60810c-eece-4cbd-a0f8-d04271924b7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 196])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: c168b27b-f467-4350-a348-4006ce85a213, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 15be01e5-1765-45f0-b521-da99b395bdb3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: d8a1310d-09c1-4844-91b7-509f93b30cfd, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 59731bd7-ffc9-493f-a33c-a1cf5ba2df30, kind: Service { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1 }, name: Name("external-dns-1eb90f9c-ba99-428e-885b-5fa2df7ae2f1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 32])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 09fdc6fd-8297-436e-903d-b18710599203, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 005b2740-84d5-473e-ae59-aa79fd2fe5b6, kind: Service { id: 09fdc6fd-8297-436e-903d-b18710599203 }, name: Name("external-dns-09fdc6fd-8297-436e-903d-b18710599203"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 6])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: c6edf68c-f558-4f51-9aed-6eb91fb3ad51, kind: Service { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52 }, name: Name("nexus-87d65a87-e1e5-43e4-9cba-b4afd5e9ac52"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 117])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6088410e-a8fc-4d33-b593-6385e863291f, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: e6898038-bf11-41c8-b971-6ff3eed6d38b, kind: Service { id: 6088410e-a8fc-4d33-b593-6385e863291f }, name: Name("nexus-6088410e-a8fc-4d33-b593-6385e863291f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 27])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 742e0330-77de-4723-a145-74751d41c73b, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: ffe6d41d-1349-42b2-a887-49d734eef422, kind: Service { id: 742e0330-77de-4723-a145-74751d41c73b }, name: Name("nexus-742e0330-77de-4723-a145-74751d41c73b"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 199, 184])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8d644923-5967-4351-a1ad-fe8a938f33e1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 740bd3e0-b30c-4033-968a-5451c8d5adcd, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 3286a4ca-fffd-4ed0-9c21-d528e1d23603, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 0b1925c8-3ae5-4372-a3de-213551574fe1, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 663bd1f7-7a29-484c-bfce-d312646b1660, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: c6d325c3-cddf-4528-8ba6-a083f6511284, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External } } } }, OmicronZoneConfig { id: c30b7bc1-fd10-4942-8055-ada48d2e3957, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External } } } }, OmicronZoneConfig { id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
10672024-07-19T17:45:48.130ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
10682024-07-19T17:45:48.130ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10692024-07-19T17:45:48.153ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
10702024-07-19T17:45:48.153ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }
10712024-07-19T17:45:48.173ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
10722024-07-19T17:45:48.173ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10732024-07-19T17:45:48.192ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
10742024-07-19T17:45:48.192ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10752024-07-19T17:45:48.212ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
10762024-07-19T17:45:48.212ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10772024-07-19T17:45:48.231ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
10782024-07-19T17:45:48.231ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10792024-07-19T17:45:48.250ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
10802024-07-19T17:45:48.250ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10812024-07-19T17:45:48.270ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
10822024-07-19T17:45:48.270ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10832024-07-19T17:45:48.290ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
10842024-07-19T17:45:48.290ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
10852024-07-19T17:45:48.311ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
10862024-07-19T17:45:48.311ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
10872024-07-19T17:45:48.333ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
10882024-07-19T17:45:48.333ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
10892024-07-19T17:45:48.356ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
10902024-07-19T17:45:48.356ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }
10912024-07-19T17:45:48.377ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
10922024-07-19T17:45:48.377ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }
10932024-07-19T17:45:48.396ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
10942024-07-19T17:45:48.396ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }
10952024-07-19T17:45:48.415ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
10962024-07-19T17:45:48.415ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }
10972024-07-19T17:45:48.435ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
10982024-07-19T17:45:48.435ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }
10992024-07-19T17:45:48.455ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
11002024-07-19T17:45:48.455ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }
11012024-07-19T17:45:48.474ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
11022024-07-19T17:45:48.474ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }
11032024-07-19T17:45:48.494ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
11042024-07-19T17:45:48.494ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }
11052024-07-19T17:45:48.514ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
11062024-07-19T17:45:48.514ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }
11072024-07-19T17:45:48.535ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
11082024-07-19T17:45:48.535ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/omicron-zones.json file = common/src/ledger.rs:177
11092024-07-19T17:45:48.535ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/omicron-zones.json file = common/src/ledger.rs:177
11102024-07-19T17:45:48.535ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
11112024-07-19T17:45:48.633ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
11122024-07-19T17:45:48.633ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
11132024-07-19T17:45:48.633ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
11142024-07-19T17:45:48.633ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
11152024-07-19T17:45:48.633ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
11162024-07-19T17:45:48.633ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
11172024-07-19T17:45:48.633ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
11182024-07-19T17:45:48.633ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
11192024-07-19T17:45:48.633ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
11202024-07-19T17:45:48.633ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
11212024-07-19T17:45:48.633ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
11222024-07-19T17:45:48.633ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
11232024-07-19T17:45:48.633ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
11242024-07-19T17:45:48.633ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
11252024-07-19T17:45:48.633ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
11262024-07-19T17:45:48.633ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
11272024-07-19T17:45:48.633ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
11282024-07-19T17:45:48.633ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
11292024-07-19T17:45:48.633ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
11302024-07-19T17:45:48.633ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660", TimeNotSynchronized), ("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1", TimeNotSynchronized), ("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9", TimeNotSynchronized), ("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203", TimeNotSynchronized), ("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1", TimeNotSynchronized), ("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb", TimeNotSynchronized), ("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957", TimeNotSynchronized), ("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52", TimeNotSynchronized), ("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec", TimeNotSynchronized), ("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284", TimeNotSynchronized), ("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17", TimeNotSynchronized), ("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7", TimeNotSynchronized), ("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603", TimeNotSynchronized), ("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f", TimeNotSynchronized), ("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1", TimeNotSynchronized), ("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b", TimeNotSynchronized), ("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b", TimeNotSynchronized), ("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1", TimeNotSynchronized), ("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 503815 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:37752 req_id = cda63b96-b096-41d3-a3ea-0f752f756964 response_code = 500 uri = /omicron-zones
11312024-07-19T17:45:48.633ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "cda63b96-b096-41d3-a3ea-0f752f756964", "content-length": "124", "date": "Fri, 19 Jul 2024 17:45:48 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "cda63b96-b096-41d3-a3ea-0f752f756964" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 233.201311ms sled_address = [fd00:1122:3344:101::1]:12345
11322024-07-19T17:45:48.869ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 8ae14f97-bd01-42ad-87d5-23eae9194521, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: bc60810c-eece-4cbd-a0f8-d04271924b7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bf3fc707-b452-4d30-a92b-e56b4d4122ee, kind: Service { id: bc60810c-eece-4cbd-a0f8-d04271924b7c }, name: Name("ntp-bc60810c-eece-4cbd-a0f8-d04271924b7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 196])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: c168b27b-f467-4350-a348-4006ce85a213, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 15be01e5-1765-45f0-b521-da99b395bdb3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: d8a1310d-09c1-4844-91b7-509f93b30cfd, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 59731bd7-ffc9-493f-a33c-a1cf5ba2df30, kind: Service { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1 }, name: Name("external-dns-1eb90f9c-ba99-428e-885b-5fa2df7ae2f1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 32])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 09fdc6fd-8297-436e-903d-b18710599203, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 005b2740-84d5-473e-ae59-aa79fd2fe5b6, kind: Service { id: 09fdc6fd-8297-436e-903d-b18710599203 }, name: Name("external-dns-09fdc6fd-8297-436e-903d-b18710599203"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 6])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: c6edf68c-f558-4f51-9aed-6eb91fb3ad51, kind: Service { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52 }, name: Name("nexus-87d65a87-e1e5-43e4-9cba-b4afd5e9ac52"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 117])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6088410e-a8fc-4d33-b593-6385e863291f, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: e6898038-bf11-41c8-b971-6ff3eed6d38b, kind: Service { id: 6088410e-a8fc-4d33-b593-6385e863291f }, name: Name("nexus-6088410e-a8fc-4d33-b593-6385e863291f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 27])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 742e0330-77de-4723-a145-74751d41c73b, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: ffe6d41d-1349-42b2-a887-49d734eef422, kind: Service { id: 742e0330-77de-4723-a145-74751d41c73b }, name: Name("nexus-742e0330-77de-4723-a145-74751d41c73b"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 199, 184])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8d644923-5967-4351-a1ad-fe8a938f33e1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 740bd3e0-b30c-4033-968a-5451c8d5adcd, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 3286a4ca-fffd-4ed0-9c21-d528e1d23603, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 0b1925c8-3ae5-4372-a3de-213551574fe1, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 663bd1f7-7a29-484c-bfce-d312646b1660, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: c6d325c3-cddf-4528-8ba6-a083f6511284, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External } } } }, OmicronZoneConfig { id: c30b7bc1-fd10-4942-8055-ada48d2e3957, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External } } } }, OmicronZoneConfig { id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
11332024-07-19T17:45:48.869ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
11342024-07-19T17:45:48.869ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11352024-07-19T17:45:48.893ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
11362024-07-19T17:45:48.893ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }
11372024-07-19T17:45:48.915ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
11382024-07-19T17:45:48.915ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11392024-07-19T17:45:48.935ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
11402024-07-19T17:45:48.935ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11412024-07-19T17:45:48.954ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
11422024-07-19T17:45:48.954ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11432024-07-19T17:45:48.973ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
11442024-07-19T17:45:48.973ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11452024-07-19T17:45:48.991ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
11462024-07-19T17:45:48.991ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11472024-07-19T17:45:49.009ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
11482024-07-19T17:45:49.009ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11492024-07-19T17:45:49.027ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
11502024-07-19T17:45:49.027ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
11512024-07-19T17:45:49.046ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
11522024-07-19T17:45:49.046ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
11532024-07-19T17:45:49.067ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
11542024-07-19T17:45:49.067ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
11552024-07-19T17:45:49.085ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
11562024-07-19T17:45:49.085ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }
11572024-07-19T17:45:49.107ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
11582024-07-19T17:45:49.107ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }
11592024-07-19T17:45:49.128ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
11602024-07-19T17:45:49.128ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }
11612024-07-19T17:45:49.147ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
11622024-07-19T17:45:49.147ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }
11632024-07-19T17:45:49.166ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
11642024-07-19T17:45:49.166ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }
11652024-07-19T17:45:49.185ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
11662024-07-19T17:45:49.185ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }
11672024-07-19T17:45:49.205ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
11682024-07-19T17:45:49.205ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }
11692024-07-19T17:45:49.225ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
11702024-07-19T17:45:49.225ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }
11712024-07-19T17:45:49.244ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
11722024-07-19T17:45:49.244ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }
11732024-07-19T17:45:49.265ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
11742024-07-19T17:45:49.265ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/omicron-zones.json file = common/src/ledger.rs:177
11752024-07-19T17:45:49.265ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/omicron-zones.json file = common/src/ledger.rs:177
11762024-07-19T17:45:49.266ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
11772024-07-19T17:45:49.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
11782024-07-19T17:45:49.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
11792024-07-19T17:45:49.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
11802024-07-19T17:45:49.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
11812024-07-19T17:45:49.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
11822024-07-19T17:45:49.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
11832024-07-19T17:45:49.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
11842024-07-19T17:45:49.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
11852024-07-19T17:45:49.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
11862024-07-19T17:45:49.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
11872024-07-19T17:45:49.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
11882024-07-19T17:45:49.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
11892024-07-19T17:45:49.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
11902024-07-19T17:45:49.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
11912024-07-19T17:45:49.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
11922024-07-19T17:45:49.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
11932024-07-19T17:45:49.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
11942024-07-19T17:45:49.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
11952024-07-19T17:45:49.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
11962024-07-19T17:45:49.361ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603", TimeNotSynchronized), ("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52", TimeNotSynchronized), ("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1", TimeNotSynchronized), ("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b", TimeNotSynchronized), ("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1", TimeNotSynchronized), ("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec", TimeNotSynchronized), ("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9", TimeNotSynchronized), ("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284", TimeNotSynchronized), ("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1", TimeNotSynchronized), ("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b", TimeNotSynchronized), ("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f", TimeNotSynchronized), ("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd", TimeNotSynchronized), ("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7", TimeNotSynchronized), ("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203", TimeNotSynchronized), ("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1", TimeNotSynchronized), ("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb", TimeNotSynchronized), ("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957", TimeNotSynchronized), ("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17", TimeNotSynchronized), ("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 492560 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:37752 req_id = 11cc6bb6-b6c3-489b-88b6-b7bf3caae8b9 response_code = 500 uri = /omicron-zones
11972024-07-19T17:45:49.362ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "11cc6bb6-b6c3-489b-88b6-b7bf3caae8b9", "content-length": "124", "date": "Fri, 19 Jul 2024 17:45:48 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "11cc6bb6-b6c3-489b-88b6-b7bf3caae8b9" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 307.64155ms sled_address = [fd00:1122:3344:101::1]:12345
11982024-07-19T17:45:49.670ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 8ae14f97-bd01-42ad-87d5-23eae9194521, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: bc60810c-eece-4cbd-a0f8-d04271924b7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bf3fc707-b452-4d30-a92b-e56b4d4122ee, kind: Service { id: bc60810c-eece-4cbd-a0f8-d04271924b7c }, name: Name("ntp-bc60810c-eece-4cbd-a0f8-d04271924b7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 196])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: c168b27b-f467-4350-a348-4006ce85a213, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 15be01e5-1765-45f0-b521-da99b395bdb3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: d8a1310d-09c1-4844-91b7-509f93b30cfd, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 59731bd7-ffc9-493f-a33c-a1cf5ba2df30, kind: Service { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1 }, name: Name("external-dns-1eb90f9c-ba99-428e-885b-5fa2df7ae2f1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 32])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 09fdc6fd-8297-436e-903d-b18710599203, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 005b2740-84d5-473e-ae59-aa79fd2fe5b6, kind: Service { id: 09fdc6fd-8297-436e-903d-b18710599203 }, name: Name("external-dns-09fdc6fd-8297-436e-903d-b18710599203"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 6])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: c6edf68c-f558-4f51-9aed-6eb91fb3ad51, kind: Service { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52 }, name: Name("nexus-87d65a87-e1e5-43e4-9cba-b4afd5e9ac52"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 117])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6088410e-a8fc-4d33-b593-6385e863291f, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: e6898038-bf11-41c8-b971-6ff3eed6d38b, kind: Service { id: 6088410e-a8fc-4d33-b593-6385e863291f }, name: Name("nexus-6088410e-a8fc-4d33-b593-6385e863291f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 27])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 742e0330-77de-4723-a145-74751d41c73b, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: ffe6d41d-1349-42b2-a887-49d734eef422, kind: Service { id: 742e0330-77de-4723-a145-74751d41c73b }, name: Name("nexus-742e0330-77de-4723-a145-74751d41c73b"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 199, 184])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8d644923-5967-4351-a1ad-fe8a938f33e1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 740bd3e0-b30c-4033-968a-5451c8d5adcd, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 3286a4ca-fffd-4ed0-9c21-d528e1d23603, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 0b1925c8-3ae5-4372-a3de-213551574fe1, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 663bd1f7-7a29-484c-bfce-d312646b1660, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: c6d325c3-cddf-4528-8ba6-a083f6511284, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External } } } }, OmicronZoneConfig { id: c30b7bc1-fd10-4942-8055-ada48d2e3957, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External } } } }, OmicronZoneConfig { id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
11992024-07-19T17:45:49.670ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
12002024-07-19T17:45:49.670ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12012024-07-19T17:45:49.694ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
12022024-07-19T17:45:49.694ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }
12032024-07-19T17:45:49.717ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
12042024-07-19T17:45:49.717ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12052024-07-19T17:45:49.740ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
12062024-07-19T17:45:49.740ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12072024-07-19T17:45:49.761ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
12082024-07-19T17:45:49.761ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12092024-07-19T17:45:49.784ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
12102024-07-19T17:45:49.784ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12112024-07-19T17:45:49.806ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
12122024-07-19T17:45:49.806ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12132024-07-19T17:45:49.826ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
12142024-07-19T17:45:49.826ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12152024-07-19T17:45:49.848ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
12162024-07-19T17:45:49.848ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
12172024-07-19T17:45:49.869ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
12182024-07-19T17:45:49.869ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
12192024-07-19T17:45:49.891ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
12202024-07-19T17:45:49.891ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
12212024-07-19T17:45:49.913ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
12222024-07-19T17:45:49.913ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }
12232024-07-19T17:45:49.935ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
12242024-07-19T17:45:49.935ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }
12252024-07-19T17:45:49.956ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
12262024-07-19T17:45:49.956ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }
12272024-07-19T17:45:49.977ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
12282024-07-19T17:45:49.977ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }
12292024-07-19T17:45:49.997ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
12302024-07-19T17:45:49.997ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }
12312024-07-19T17:45:50.018ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
12322024-07-19T17:45:50.018ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }
12332024-07-19T17:45:50.041ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
12342024-07-19T17:45:50.041ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }
12352024-07-19T17:45:50.062ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
12362024-07-19T17:45:50.062ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }
12372024-07-19T17:45:50.084ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
12382024-07-19T17:45:50.084ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }
12392024-07-19T17:45:50.105ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
12402024-07-19T17:45:50.105ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/omicron-zones.json file = common/src/ledger.rs:177
12412024-07-19T17:45:50.105ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/omicron-zones.json file = common/src/ledger.rs:177
12422024-07-19T17:45:50.105ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
12432024-07-19T17:45:50.206ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
12442024-07-19T17:45:50.206ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
12452024-07-19T17:45:50.206ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
12462024-07-19T17:45:50.206ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
12472024-07-19T17:45:50.206ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
12482024-07-19T17:45:50.206ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
12492024-07-19T17:45:50.206ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
12502024-07-19T17:45:50.206ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
12512024-07-19T17:45:50.206ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
12522024-07-19T17:45:50.206ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
12532024-07-19T17:45:50.206ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
12542024-07-19T17:45:50.206ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
12552024-07-19T17:45:50.206ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
12562024-07-19T17:45:50.206ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
12572024-07-19T17:45:50.206ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
12582024-07-19T17:45:50.206ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
12592024-07-19T17:45:50.206ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
12602024-07-19T17:45:50.206ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
12612024-07-19T17:45:50.206ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
12622024-07-19T17:45:50.206ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1", TimeNotSynchronized), ("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1", TimeNotSynchronized), ("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd", TimeNotSynchronized), ("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203", TimeNotSynchronized), ("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603", TimeNotSynchronized), ("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9", TimeNotSynchronized), ("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec", TimeNotSynchronized), ("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660", TimeNotSynchronized), ("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb", TimeNotSynchronized), ("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b", TimeNotSynchronized), ("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957", TimeNotSynchronized), ("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17", TimeNotSynchronized), ("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b", TimeNotSynchronized), ("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1", TimeNotSynchronized), ("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1", TimeNotSynchronized), ("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284", TimeNotSynchronized), ("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52", TimeNotSynchronized), ("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7", TimeNotSynchronized), ("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 535807 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:37752 req_id = 5384f9aa-45a9-4a9c-8f3e-4cb8308fef4d response_code = 500 uri = /omicron-zones
12632024-07-19T17:45:50.206ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "5384f9aa-45a9-4a9c-8f3e-4cb8308fef4d", "content-length": "124", "date": "Fri, 19 Jul 2024 17:45:49 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "5384f9aa-45a9-4a9c-8f3e-4cb8308fef4d" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 311.250186ms sled_address = [fd00:1122:3344:101::1]:12345
12642024-07-19T17:45:50.519ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 8ae14f97-bd01-42ad-87d5-23eae9194521, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: bc60810c-eece-4cbd-a0f8-d04271924b7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bf3fc707-b452-4d30-a92b-e56b4d4122ee, kind: Service { id: bc60810c-eece-4cbd-a0f8-d04271924b7c }, name: Name("ntp-bc60810c-eece-4cbd-a0f8-d04271924b7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 196])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: c168b27b-f467-4350-a348-4006ce85a213, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 15be01e5-1765-45f0-b521-da99b395bdb3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: d8a1310d-09c1-4844-91b7-509f93b30cfd, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 59731bd7-ffc9-493f-a33c-a1cf5ba2df30, kind: Service { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1 }, name: Name("external-dns-1eb90f9c-ba99-428e-885b-5fa2df7ae2f1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 32])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 09fdc6fd-8297-436e-903d-b18710599203, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 005b2740-84d5-473e-ae59-aa79fd2fe5b6, kind: Service { id: 09fdc6fd-8297-436e-903d-b18710599203 }, name: Name("external-dns-09fdc6fd-8297-436e-903d-b18710599203"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 6])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: c6edf68c-f558-4f51-9aed-6eb91fb3ad51, kind: Service { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52 }, name: Name("nexus-87d65a87-e1e5-43e4-9cba-b4afd5e9ac52"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 117])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6088410e-a8fc-4d33-b593-6385e863291f, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: e6898038-bf11-41c8-b971-6ff3eed6d38b, kind: Service { id: 6088410e-a8fc-4d33-b593-6385e863291f }, name: Name("nexus-6088410e-a8fc-4d33-b593-6385e863291f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 27])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 742e0330-77de-4723-a145-74751d41c73b, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: ffe6d41d-1349-42b2-a887-49d734eef422, kind: Service { id: 742e0330-77de-4723-a145-74751d41c73b }, name: Name("nexus-742e0330-77de-4723-a145-74751d41c73b"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 199, 184])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8d644923-5967-4351-a1ad-fe8a938f33e1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 740bd3e0-b30c-4033-968a-5451c8d5adcd, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 3286a4ca-fffd-4ed0-9c21-d528e1d23603, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 0b1925c8-3ae5-4372-a3de-213551574fe1, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 663bd1f7-7a29-484c-bfce-d312646b1660, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: c6d325c3-cddf-4528-8ba6-a083f6511284, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External } } } }, OmicronZoneConfig { id: c30b7bc1-fd10-4942-8055-ada48d2e3957, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External } } } }, OmicronZoneConfig { id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
12652024-07-19T17:45:50.520ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
12662024-07-19T17:45:50.520ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12672024-07-19T17:45:50.544ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
12682024-07-19T17:45:50.544ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }
12692024-07-19T17:45:50.566ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
12702024-07-19T17:45:50.566ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12712024-07-19T17:45:50.584ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
12722024-07-19T17:45:50.584ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12732024-07-19T17:45:50.601ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
12742024-07-19T17:45:50.601ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12752024-07-19T17:45:50.618ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
12762024-07-19T17:45:50.619ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12772024-07-19T17:45:50.637ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
12782024-07-19T17:45:50.637ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12792024-07-19T17:45:50.659ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
12802024-07-19T17:45:50.659ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12812024-07-19T17:45:50.680ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
12822024-07-19T17:45:50.680ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
12832024-07-19T17:45:50.700ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
12842024-07-19T17:45:50.700ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
12852024-07-19T17:45:50.722ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
12862024-07-19T17:45:50.722ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
12872024-07-19T17:45:50.745ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
12882024-07-19T17:45:50.745ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }
12892024-07-19T17:45:50.769ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
12902024-07-19T17:45:50.769ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }
12912024-07-19T17:45:50.789ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
12922024-07-19T17:45:50.789ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }
12932024-07-19T17:45:50.808ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
12942024-07-19T17:45:50.808ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }
12952024-07-19T17:45:50.832ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
12962024-07-19T17:45:50.832ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }
12972024-07-19T17:45:50.853ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
12982024-07-19T17:45:50.853ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }
12992024-07-19T17:45:50.873ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
13002024-07-19T17:45:50.873ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }
13012024-07-19T17:45:50.894ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
13022024-07-19T17:45:50.894ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }
13032024-07-19T17:45:50.915ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
13042024-07-19T17:45:50.915ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }
13052024-07-19T17:45:50.935ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
13062024-07-19T17:45:50.935ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/omicron-zones.json file = common/src/ledger.rs:177
13072024-07-19T17:45:50.935ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/omicron-zones.json file = common/src/ledger.rs:177
13082024-07-19T17:45:50.936ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
13092024-07-19T17:45:51.034ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
13102024-07-19T17:45:51.034ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
13112024-07-19T17:45:51.034ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
13122024-07-19T17:45:51.034ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
13132024-07-19T17:45:51.034ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
13142024-07-19T17:45:51.034ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
13152024-07-19T17:45:51.034ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
13162024-07-19T17:45:51.034ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
13172024-07-19T17:45:51.034ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
13182024-07-19T17:45:51.034ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
13192024-07-19T17:45:51.034ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
13202024-07-19T17:45:51.034ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
13212024-07-19T17:45:51.034ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
13222024-07-19T17:45:51.034ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
13232024-07-19T17:45:51.034ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
13242024-07-19T17:45:51.034ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
13252024-07-19T17:45:51.034ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
13262024-07-19T17:45:51.034ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
13272024-07-19T17:45:51.034ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
13282024-07-19T17:45:51.034ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1", TimeNotSynchronized), ("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b", TimeNotSynchronized), ("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd", TimeNotSynchronized), ("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1", TimeNotSynchronized), ("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9", TimeNotSynchronized), ("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7", TimeNotSynchronized), ("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17", TimeNotSynchronized), ("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603", TimeNotSynchronized), ("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f", TimeNotSynchronized), ("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b", TimeNotSynchronized), ("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1", TimeNotSynchronized), ("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660", TimeNotSynchronized), ("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb", TimeNotSynchronized), ("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203", TimeNotSynchronized), ("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec", TimeNotSynchronized), ("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957", TimeNotSynchronized), ("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284", TimeNotSynchronized), ("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52", TimeNotSynchronized), ("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 515402 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:37752 req_id = 5a03f9ca-d7b8-4beb-bc7b-6fc87f940456 response_code = 500 uri = /omicron-zones
13292024-07-19T17:45:51.034ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "5a03f9ca-d7b8-4beb-bc7b-6fc87f940456", "content-length": "124", "date": "Fri, 19 Jul 2024 17:45:51 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "5a03f9ca-d7b8-4beb-bc7b-6fc87f940456" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 278.688335ms sled_address = [fd00:1122:3344:101::1]:12345
13302024-07-19T17:45:51.314ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 8ae14f97-bd01-42ad-87d5-23eae9194521, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: bc60810c-eece-4cbd-a0f8-d04271924b7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bf3fc707-b452-4d30-a92b-e56b4d4122ee, kind: Service { id: bc60810c-eece-4cbd-a0f8-d04271924b7c }, name: Name("ntp-bc60810c-eece-4cbd-a0f8-d04271924b7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 196])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: c168b27b-f467-4350-a348-4006ce85a213, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 15be01e5-1765-45f0-b521-da99b395bdb3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: d8a1310d-09c1-4844-91b7-509f93b30cfd, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 59731bd7-ffc9-493f-a33c-a1cf5ba2df30, kind: Service { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1 }, name: Name("external-dns-1eb90f9c-ba99-428e-885b-5fa2df7ae2f1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 32])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 09fdc6fd-8297-436e-903d-b18710599203, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 005b2740-84d5-473e-ae59-aa79fd2fe5b6, kind: Service { id: 09fdc6fd-8297-436e-903d-b18710599203 }, name: Name("external-dns-09fdc6fd-8297-436e-903d-b18710599203"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 6])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: c6edf68c-f558-4f51-9aed-6eb91fb3ad51, kind: Service { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52 }, name: Name("nexus-87d65a87-e1e5-43e4-9cba-b4afd5e9ac52"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 117])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6088410e-a8fc-4d33-b593-6385e863291f, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: e6898038-bf11-41c8-b971-6ff3eed6d38b, kind: Service { id: 6088410e-a8fc-4d33-b593-6385e863291f }, name: Name("nexus-6088410e-a8fc-4d33-b593-6385e863291f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 27])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 742e0330-77de-4723-a145-74751d41c73b, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: ffe6d41d-1349-42b2-a887-49d734eef422, kind: Service { id: 742e0330-77de-4723-a145-74751d41c73b }, name: Name("nexus-742e0330-77de-4723-a145-74751d41c73b"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 199, 184])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8d644923-5967-4351-a1ad-fe8a938f33e1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 740bd3e0-b30c-4033-968a-5451c8d5adcd, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 3286a4ca-fffd-4ed0-9c21-d528e1d23603, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 0b1925c8-3ae5-4372-a3de-213551574fe1, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 663bd1f7-7a29-484c-bfce-d312646b1660, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: c6d325c3-cddf-4528-8ba6-a083f6511284, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External } } } }, OmicronZoneConfig { id: c30b7bc1-fd10-4942-8055-ada48d2e3957, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External } } } }, OmicronZoneConfig { id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
13312024-07-19T17:45:51.315ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
13322024-07-19T17:45:51.315ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13332024-07-19T17:45:51.337ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
13342024-07-19T17:45:51.338ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }
13352024-07-19T17:45:51.359ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
13362024-07-19T17:45:51.359ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13372024-07-19T17:45:51.380ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
13382024-07-19T17:45:51.380ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13392024-07-19T17:45:51.399ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
13402024-07-19T17:45:51.399ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13412024-07-19T17:45:51.418ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
13422024-07-19T17:45:51.418ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13432024-07-19T17:45:51.440ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
13442024-07-19T17:45:51.440ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13452024-07-19T17:45:51.460ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
13462024-07-19T17:45:51.460ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13472024-07-19T17:45:51.482ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
13482024-07-19T17:45:51.482ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
13492024-07-19T17:45:51.503ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
13502024-07-19T17:45:51.503ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
13512024-07-19T17:45:51.524ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
13522024-07-19T17:45:51.524ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
13532024-07-19T17:45:51.543ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
13542024-07-19T17:45:51.543ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }
13552024-07-19T17:45:51.563ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
13562024-07-19T17:45:51.563ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }
13572024-07-19T17:45:51.584ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
13582024-07-19T17:45:51.584ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }
13592024-07-19T17:45:51.604ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
13602024-07-19T17:45:51.604ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }
13612024-07-19T17:45:51.625ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
13622024-07-19T17:45:51.625ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }
13632024-07-19T17:45:51.643ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
13642024-07-19T17:45:51.643ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }
13652024-07-19T17:45:51.663ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
13662024-07-19T17:45:51.663ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }
13672024-07-19T17:45:51.683ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
13682024-07-19T17:45:51.683ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }
13692024-07-19T17:45:51.704ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
13702024-07-19T17:45:51.704ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }
13712024-07-19T17:45:51.724ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
13722024-07-19T17:45:51.724ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/omicron-zones.json file = common/src/ledger.rs:177
13732024-07-19T17:45:51.724ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/omicron-zones.json file = common/src/ledger.rs:177
13742024-07-19T17:45:51.724ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
13752024-07-19T17:45:51.823ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
13762024-07-19T17:45:51.823ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
13772024-07-19T17:45:51.823ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
13782024-07-19T17:45:51.823ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
13792024-07-19T17:45:51.823ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
13802024-07-19T17:45:51.823ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
13812024-07-19T17:45:51.823ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
13822024-07-19T17:45:51.823ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
13832024-07-19T17:45:51.823ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
13842024-07-19T17:45:51.823ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
13852024-07-19T17:45:51.823ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
13862024-07-19T17:45:51.823ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
13872024-07-19T17:45:51.823ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
13882024-07-19T17:45:51.823ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
13892024-07-19T17:45:51.823ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
13902024-07-19T17:45:51.823ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
13912024-07-19T17:45:51.823ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
13922024-07-19T17:45:51.823ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
13932024-07-19T17:45:51.823ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
13942024-07-19T17:45:51.823ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b", TimeNotSynchronized), ("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7", TimeNotSynchronized), ("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1", TimeNotSynchronized), ("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1", TimeNotSynchronized), ("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660", TimeNotSynchronized), ("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17", TimeNotSynchronized), ("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52", TimeNotSynchronized), ("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603", TimeNotSynchronized), ("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b", TimeNotSynchronized), ("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9", TimeNotSynchronized), ("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb", TimeNotSynchronized), ("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957", TimeNotSynchronized), ("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1", TimeNotSynchronized), ("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1", TimeNotSynchronized), ("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284", TimeNotSynchronized), ("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec", TimeNotSynchronized), ("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd", TimeNotSynchronized), ("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203", TimeNotSynchronized), ("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 509135 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:37752 req_id = 991bf31d-0c0a-4434-a880-b81a79a04cf4 response_code = 500 uri = /omicron-zones
13952024-07-19T17:45:51.823ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "991bf31d-0c0a-4434-a880-b81a79a04cf4", "content-length": "124", "date": "Fri, 19 Jul 2024 17:45:51 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "991bf31d-0c0a-4434-a880-b81a79a04cf4" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 442.383338ms sled_address = [fd00:1122:3344:101::1]:12345
13962024-07-19T17:45:52.268ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 8ae14f97-bd01-42ad-87d5-23eae9194521, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: bc60810c-eece-4cbd-a0f8-d04271924b7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bf3fc707-b452-4d30-a92b-e56b4d4122ee, kind: Service { id: bc60810c-eece-4cbd-a0f8-d04271924b7c }, name: Name("ntp-bc60810c-eece-4cbd-a0f8-d04271924b7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 196])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: c168b27b-f467-4350-a348-4006ce85a213, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 15be01e5-1765-45f0-b521-da99b395bdb3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: d8a1310d-09c1-4844-91b7-509f93b30cfd, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 59731bd7-ffc9-493f-a33c-a1cf5ba2df30, kind: Service { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1 }, name: Name("external-dns-1eb90f9c-ba99-428e-885b-5fa2df7ae2f1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 32])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 09fdc6fd-8297-436e-903d-b18710599203, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 005b2740-84d5-473e-ae59-aa79fd2fe5b6, kind: Service { id: 09fdc6fd-8297-436e-903d-b18710599203 }, name: Name("external-dns-09fdc6fd-8297-436e-903d-b18710599203"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 6])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: c6edf68c-f558-4f51-9aed-6eb91fb3ad51, kind: Service { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52 }, name: Name("nexus-87d65a87-e1e5-43e4-9cba-b4afd5e9ac52"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 117])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6088410e-a8fc-4d33-b593-6385e863291f, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: e6898038-bf11-41c8-b971-6ff3eed6d38b, kind: Service { id: 6088410e-a8fc-4d33-b593-6385e863291f }, name: Name("nexus-6088410e-a8fc-4d33-b593-6385e863291f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 27])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 742e0330-77de-4723-a145-74751d41c73b, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: ffe6d41d-1349-42b2-a887-49d734eef422, kind: Service { id: 742e0330-77de-4723-a145-74751d41c73b }, name: Name("nexus-742e0330-77de-4723-a145-74751d41c73b"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 199, 184])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8d644923-5967-4351-a1ad-fe8a938f33e1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 740bd3e0-b30c-4033-968a-5451c8d5adcd, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 3286a4ca-fffd-4ed0-9c21-d528e1d23603, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 0b1925c8-3ae5-4372-a3de-213551574fe1, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 663bd1f7-7a29-484c-bfce-d312646b1660, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: c6d325c3-cddf-4528-8ba6-a083f6511284, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External } } } }, OmicronZoneConfig { id: c30b7bc1-fd10-4942-8055-ada48d2e3957, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External } } } }, OmicronZoneConfig { id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
13972024-07-19T17:45:52.268ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
13982024-07-19T17:45:52.268ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13992024-07-19T17:45:52.292ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
14002024-07-19T17:45:52.292ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }
14012024-07-19T17:45:52.316ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
14022024-07-19T17:45:52.316ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14032024-07-19T17:45:52.335ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
14042024-07-19T17:45:52.335ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14052024-07-19T17:45:52.353ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
14062024-07-19T17:45:52.353ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14072024-07-19T17:45:52.372ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
14082024-07-19T17:45:52.372ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14092024-07-19T17:45:52.394ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
14102024-07-19T17:45:52.394ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14112024-07-19T17:45:52.415ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
14122024-07-19T17:45:52.415ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14132024-07-19T17:45:52.435ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
14142024-07-19T17:45:52.435ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
14152024-07-19T17:45:52.455ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
14162024-07-19T17:45:52.455ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
14172024-07-19T17:45:52.476ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
14182024-07-19T17:45:52.476ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
14192024-07-19T17:45:52.497ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
14202024-07-19T17:45:52.497ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }
14212024-07-19T17:45:52.517ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
14222024-07-19T17:45:52.517ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }
14232024-07-19T17:45:52.538ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
14242024-07-19T17:45:52.538ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }
14252024-07-19T17:45:52.559ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
14262024-07-19T17:45:52.559ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }
14272024-07-19T17:45:52.578ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
14282024-07-19T17:45:52.579ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }
14292024-07-19T17:45:52.599ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
14302024-07-19T17:45:52.599ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }
14312024-07-19T17:45:52.621ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
14322024-07-19T17:45:52.621ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }
14332024-07-19T17:45:52.642ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
14342024-07-19T17:45:52.642ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }
14352024-07-19T17:45:52.662ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
14362024-07-19T17:45:52.662ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }
14372024-07-19T17:45:52.682ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
14382024-07-19T17:45:52.682ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/omicron-zones.json file = common/src/ledger.rs:177
14392024-07-19T17:45:52.683ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/omicron-zones.json file = common/src/ledger.rs:177
14402024-07-19T17:45:52.683ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
14412024-07-19T17:45:52.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
14422024-07-19T17:45:52.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
14432024-07-19T17:45:52.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
14442024-07-19T17:45:52.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
14452024-07-19T17:45:52.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
14462024-07-19T17:45:52.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
14472024-07-19T17:45:52.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
14482024-07-19T17:45:52.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
14492024-07-19T17:45:52.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
14502024-07-19T17:45:52.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
14512024-07-19T17:45:52.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
14522024-07-19T17:45:52.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
14532024-07-19T17:45:52.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
14542024-07-19T17:45:52.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
14552024-07-19T17:45:52.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
14562024-07-19T17:45:52.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
14572024-07-19T17:45:52.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
14582024-07-19T17:45:52.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
14592024-07-19T17:45:52.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
14602024-07-19T17:45:52.783ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203", TimeNotSynchronized), ("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52", TimeNotSynchronized), ("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec", TimeNotSynchronized), ("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7", TimeNotSynchronized), ("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b", TimeNotSynchronized), ("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb", TimeNotSynchronized), ("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17", TimeNotSynchronized), ("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f", TimeNotSynchronized), ("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957", TimeNotSynchronized), ("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284", TimeNotSynchronized), ("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b", TimeNotSynchronized), ("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603", TimeNotSynchronized), ("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1", TimeNotSynchronized), ("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd", TimeNotSynchronized), ("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1", TimeNotSynchronized), ("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660", TimeNotSynchronized), ("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1", TimeNotSynchronized), ("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9", TimeNotSynchronized), ("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 514708 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:37752 req_id = eea56224-6a5a-49e3-a37a-cdb7c0f1304d response_code = 500 uri = /omicron-zones
14612024-07-19T17:45:52.783ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "eea56224-6a5a-49e3-a37a-cdb7c0f1304d", "content-length": "124", "date": "Fri, 19 Jul 2024 17:45:52 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "eea56224-6a5a-49e3-a37a-cdb7c0f1304d" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 305.842624ms sled_address = [fd00:1122:3344:101::1]:12345
14622024-07-19T17:45:53.090ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 8ae14f97-bd01-42ad-87d5-23eae9194521, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: bc60810c-eece-4cbd-a0f8-d04271924b7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bf3fc707-b452-4d30-a92b-e56b4d4122ee, kind: Service { id: bc60810c-eece-4cbd-a0f8-d04271924b7c }, name: Name("ntp-bc60810c-eece-4cbd-a0f8-d04271924b7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 196])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: c168b27b-f467-4350-a348-4006ce85a213, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 15be01e5-1765-45f0-b521-da99b395bdb3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: d8a1310d-09c1-4844-91b7-509f93b30cfd, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 59731bd7-ffc9-493f-a33c-a1cf5ba2df30, kind: Service { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1 }, name: Name("external-dns-1eb90f9c-ba99-428e-885b-5fa2df7ae2f1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 32])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 09fdc6fd-8297-436e-903d-b18710599203, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 005b2740-84d5-473e-ae59-aa79fd2fe5b6, kind: Service { id: 09fdc6fd-8297-436e-903d-b18710599203 }, name: Name("external-dns-09fdc6fd-8297-436e-903d-b18710599203"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 6])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: c6edf68c-f558-4f51-9aed-6eb91fb3ad51, kind: Service { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52 }, name: Name("nexus-87d65a87-e1e5-43e4-9cba-b4afd5e9ac52"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 117])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6088410e-a8fc-4d33-b593-6385e863291f, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: e6898038-bf11-41c8-b971-6ff3eed6d38b, kind: Service { id: 6088410e-a8fc-4d33-b593-6385e863291f }, name: Name("nexus-6088410e-a8fc-4d33-b593-6385e863291f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 27])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 742e0330-77de-4723-a145-74751d41c73b, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: ffe6d41d-1349-42b2-a887-49d734eef422, kind: Service { id: 742e0330-77de-4723-a145-74751d41c73b }, name: Name("nexus-742e0330-77de-4723-a145-74751d41c73b"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 199, 184])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8d644923-5967-4351-a1ad-fe8a938f33e1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 740bd3e0-b30c-4033-968a-5451c8d5adcd, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 3286a4ca-fffd-4ed0-9c21-d528e1d23603, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 0b1925c8-3ae5-4372-a3de-213551574fe1, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 663bd1f7-7a29-484c-bfce-d312646b1660, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: c6d325c3-cddf-4528-8ba6-a083f6511284, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External } } } }, OmicronZoneConfig { id: c30b7bc1-fd10-4942-8055-ada48d2e3957, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External } } } }, OmicronZoneConfig { id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
14632024-07-19T17:45:53.091ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
14642024-07-19T17:45:53.091ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14652024-07-19T17:45:53.115ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
14662024-07-19T17:45:53.115ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }
14672024-07-19T17:45:53.135ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
14682024-07-19T17:45:53.135ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14692024-07-19T17:45:53.153ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
14702024-07-19T17:45:53.153ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14712024-07-19T17:45:53.170ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
14722024-07-19T17:45:53.170ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14732024-07-19T17:45:53.190ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
14742024-07-19T17:45:53.190ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14752024-07-19T17:45:53.209ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
14762024-07-19T17:45:53.209ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14772024-07-19T17:45:53.229ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
14782024-07-19T17:45:53.229ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14792024-07-19T17:45:53.247ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
14802024-07-19T17:45:53.248ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
14812024-07-19T17:45:53.265ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
14822024-07-19T17:45:53.265ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
14832024-07-19T17:45:53.285ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
14842024-07-19T17:45:53.285ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
14852024-07-19T17:45:53.306ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
14862024-07-19T17:45:53.306ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }
14872024-07-19T17:45:53.326ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
14882024-07-19T17:45:53.327ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }
14892024-07-19T17:45:53.344ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
14902024-07-19T17:45:53.344ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }
14912024-07-19T17:45:53.364ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
14922024-07-19T17:45:53.364ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }
14932024-07-19T17:45:53.381ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
14942024-07-19T17:45:53.381ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }
14952024-07-19T17:45:53.402ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
14962024-07-19T17:45:53.402ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }
14972024-07-19T17:45:53.424ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
14982024-07-19T17:45:53.424ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }
14992024-07-19T17:45:53.445ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
15002024-07-19T17:45:53.445ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }
15012024-07-19T17:45:53.465ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
15022024-07-19T17:45:53.465ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }
15032024-07-19T17:45:53.485ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
15042024-07-19T17:45:53.485ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/omicron-zones.json file = common/src/ledger.rs:177
15052024-07-19T17:45:53.485ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/omicron-zones.json file = common/src/ledger.rs:177
15062024-07-19T17:45:53.485ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
15072024-07-19T17:45:53.583ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
15082024-07-19T17:45:53.583ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
15092024-07-19T17:45:53.583ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
15102024-07-19T17:45:53.583ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
15112024-07-19T17:45:53.583ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
15122024-07-19T17:45:53.583ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
15132024-07-19T17:45:53.583ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
15142024-07-19T17:45:53.583ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
15152024-07-19T17:45:53.583ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
15162024-07-19T17:45:53.583ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
15172024-07-19T17:45:53.583ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
15182024-07-19T17:45:53.583ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
15192024-07-19T17:45:53.583ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
15202024-07-19T17:45:53.583ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
15212024-07-19T17:45:53.583ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
15222024-07-19T17:45:53.583ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
15232024-07-19T17:45:53.583ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
15242024-07-19T17:45:53.583ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
15252024-07-19T17:45:53.583ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
15262024-07-19T17:45:53.583ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9", TimeNotSynchronized), ("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb", TimeNotSynchronized), ("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17", TimeNotSynchronized), ("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284", TimeNotSynchronized), ("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52", TimeNotSynchronized), ("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203", TimeNotSynchronized), ("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7", TimeNotSynchronized), ("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957", TimeNotSynchronized), ("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f", TimeNotSynchronized), ("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b", TimeNotSynchronized), ("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b", TimeNotSynchronized), ("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1", TimeNotSynchronized), ("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1", TimeNotSynchronized), ("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd", TimeNotSynchronized), ("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660", TimeNotSynchronized), ("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603", TimeNotSynchronized), ("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec", TimeNotSynchronized), ("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1", TimeNotSynchronized), ("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 492626 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:37752 req_id = 3b5b5885-01d3-45f6-abe4-784a6e1231c1 response_code = 500 uri = /omicron-zones
15272024-07-19T17:45:53.583ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "3b5b5885-01d3-45f6-abe4-784a6e1231c1", "content-length": "124", "date": "Fri, 19 Jul 2024 17:45:53 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "3b5b5885-01d3-45f6-abe4-784a6e1231c1" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 357.028805ms sled_address = [fd00:1122:3344:101::1]:12345
15282024-07-19T17:45:53.941ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 8ae14f97-bd01-42ad-87d5-23eae9194521, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: bc60810c-eece-4cbd-a0f8-d04271924b7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bf3fc707-b452-4d30-a92b-e56b4d4122ee, kind: Service { id: bc60810c-eece-4cbd-a0f8-d04271924b7c }, name: Name("ntp-bc60810c-eece-4cbd-a0f8-d04271924b7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 196])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: c168b27b-f467-4350-a348-4006ce85a213, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 15be01e5-1765-45f0-b521-da99b395bdb3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: d8a1310d-09c1-4844-91b7-509f93b30cfd, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 59731bd7-ffc9-493f-a33c-a1cf5ba2df30, kind: Service { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1 }, name: Name("external-dns-1eb90f9c-ba99-428e-885b-5fa2df7ae2f1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 32])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 09fdc6fd-8297-436e-903d-b18710599203, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 005b2740-84d5-473e-ae59-aa79fd2fe5b6, kind: Service { id: 09fdc6fd-8297-436e-903d-b18710599203 }, name: Name("external-dns-09fdc6fd-8297-436e-903d-b18710599203"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 6])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: c6edf68c-f558-4f51-9aed-6eb91fb3ad51, kind: Service { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52 }, name: Name("nexus-87d65a87-e1e5-43e4-9cba-b4afd5e9ac52"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 117])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6088410e-a8fc-4d33-b593-6385e863291f, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: e6898038-bf11-41c8-b971-6ff3eed6d38b, kind: Service { id: 6088410e-a8fc-4d33-b593-6385e863291f }, name: Name("nexus-6088410e-a8fc-4d33-b593-6385e863291f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 27])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 742e0330-77de-4723-a145-74751d41c73b, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: ffe6d41d-1349-42b2-a887-49d734eef422, kind: Service { id: 742e0330-77de-4723-a145-74751d41c73b }, name: Name("nexus-742e0330-77de-4723-a145-74751d41c73b"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 199, 184])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8d644923-5967-4351-a1ad-fe8a938f33e1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 740bd3e0-b30c-4033-968a-5451c8d5adcd, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 3286a4ca-fffd-4ed0-9c21-d528e1d23603, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 0b1925c8-3ae5-4372-a3de-213551574fe1, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 663bd1f7-7a29-484c-bfce-d312646b1660, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: c6d325c3-cddf-4528-8ba6-a083f6511284, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External } } } }, OmicronZoneConfig { id: c30b7bc1-fd10-4942-8055-ada48d2e3957, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External } } } }, OmicronZoneConfig { id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
15292024-07-19T17:45:53.941ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
15302024-07-19T17:45:53.941ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15312024-07-19T17:45:53.965ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
15322024-07-19T17:45:53.965ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }
15332024-07-19T17:45:53.986ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
15342024-07-19T17:45:53.986ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15352024-07-19T17:45:54.007ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
15362024-07-19T17:45:54.007ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15372024-07-19T17:45:54.032ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
15382024-07-19T17:45:54.032ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15392024-07-19T17:45:54.054ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
15402024-07-19T17:45:54.054ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15412024-07-19T17:45:54.077ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
15422024-07-19T17:45:54.078ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15432024-07-19T17:45:54.099ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
15442024-07-19T17:45:54.099ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15452024-07-19T17:45:54.119ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
15462024-07-19T17:45:54.119ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
15472024-07-19T17:45:54.138ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
15482024-07-19T17:45:54.138ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
15492024-07-19T17:45:54.160ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
15502024-07-19T17:45:54.160ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
15512024-07-19T17:45:54.180ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
15522024-07-19T17:45:54.181ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }
15532024-07-19T17:45:54.200ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
15542024-07-19T17:45:54.200ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }
15552024-07-19T17:45:54.221ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
15562024-07-19T17:45:54.221ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }
15572024-07-19T17:45:54.243ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
15582024-07-19T17:45:54.243ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }
15592024-07-19T17:45:54.262ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
15602024-07-19T17:45:54.262ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }
15612024-07-19T17:45:54.282ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
15622024-07-19T17:45:54.282ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }
15632024-07-19T17:45:54.303ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
15642024-07-19T17:45:54.303ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }
15652024-07-19T17:45:54.325ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
15662024-07-19T17:45:54.325ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }
15672024-07-19T17:45:54.349ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
15682024-07-19T17:45:54.349ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }
15692024-07-19T17:45:54.370ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
15702024-07-19T17:45:54.370ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/omicron-zones.json file = common/src/ledger.rs:177
15712024-07-19T17:45:54.370ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/omicron-zones.json file = common/src/ledger.rs:177
15722024-07-19T17:45:54.370ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
15732024-07-19T17:45:54.472ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
15742024-07-19T17:45:54.472ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
15752024-07-19T17:45:54.472ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
15762024-07-19T17:45:54.472ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
15772024-07-19T17:45:54.472ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
15782024-07-19T17:45:54.472ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
15792024-07-19T17:45:54.472ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
15802024-07-19T17:45:54.472ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
15812024-07-19T17:45:54.472ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
15822024-07-19T17:45:54.472ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
15832024-07-19T17:45:54.472ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
15842024-07-19T17:45:54.472ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
15852024-07-19T17:45:54.472ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
15862024-07-19T17:45:54.472ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
15872024-07-19T17:45:54.472ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
15882024-07-19T17:45:54.472ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
15892024-07-19T17:45:54.472ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
15902024-07-19T17:45:54.472ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
15912024-07-19T17:45:54.472ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
15922024-07-19T17:45:54.472ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec", TimeNotSynchronized), ("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7", TimeNotSynchronized), ("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17", TimeNotSynchronized), ("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9", TimeNotSynchronized), ("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52", TimeNotSynchronized), ("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660", TimeNotSynchronized), ("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1", TimeNotSynchronized), ("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f", TimeNotSynchronized), ("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603", TimeNotSynchronized), ("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd", TimeNotSynchronized), ("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1", TimeNotSynchronized), ("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1", TimeNotSynchronized), ("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1", TimeNotSynchronized), ("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b", TimeNotSynchronized), ("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b", TimeNotSynchronized), ("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284", TimeNotSynchronized), ("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb", TimeNotSynchronized), ("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957", TimeNotSynchronized), ("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 530795 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:37752 req_id = 7497e40f-6825-4867-a306-87661587dc61 response_code = 500 uri = /omicron-zones
15932024-07-19T17:45:54.472ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "7497e40f-6825-4867-a306-87661587dc61", "content-length": "124", "date": "Fri, 19 Jul 2024 17:45:54 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "7497e40f-6825-4867-a306-87661587dc61" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 709.831138ms sled_address = [fd00:1122:3344:101::1]:12345
15942024-07-19T17:45:55.184ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 8ae14f97-bd01-42ad-87d5-23eae9194521, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: bc60810c-eece-4cbd-a0f8-d04271924b7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bf3fc707-b452-4d30-a92b-e56b4d4122ee, kind: Service { id: bc60810c-eece-4cbd-a0f8-d04271924b7c }, name: Name("ntp-bc60810c-eece-4cbd-a0f8-d04271924b7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 196])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: c168b27b-f467-4350-a348-4006ce85a213, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 15be01e5-1765-45f0-b521-da99b395bdb3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: d8a1310d-09c1-4844-91b7-509f93b30cfd, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 59731bd7-ffc9-493f-a33c-a1cf5ba2df30, kind: Service { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1 }, name: Name("external-dns-1eb90f9c-ba99-428e-885b-5fa2df7ae2f1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 32])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 09fdc6fd-8297-436e-903d-b18710599203, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 005b2740-84d5-473e-ae59-aa79fd2fe5b6, kind: Service { id: 09fdc6fd-8297-436e-903d-b18710599203 }, name: Name("external-dns-09fdc6fd-8297-436e-903d-b18710599203"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 6])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: c6edf68c-f558-4f51-9aed-6eb91fb3ad51, kind: Service { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52 }, name: Name("nexus-87d65a87-e1e5-43e4-9cba-b4afd5e9ac52"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 117])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6088410e-a8fc-4d33-b593-6385e863291f, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: e6898038-bf11-41c8-b971-6ff3eed6d38b, kind: Service { id: 6088410e-a8fc-4d33-b593-6385e863291f }, name: Name("nexus-6088410e-a8fc-4d33-b593-6385e863291f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 27])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 742e0330-77de-4723-a145-74751d41c73b, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: ffe6d41d-1349-42b2-a887-49d734eef422, kind: Service { id: 742e0330-77de-4723-a145-74751d41c73b }, name: Name("nexus-742e0330-77de-4723-a145-74751d41c73b"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 199, 184])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8d644923-5967-4351-a1ad-fe8a938f33e1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 740bd3e0-b30c-4033-968a-5451c8d5adcd, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 3286a4ca-fffd-4ed0-9c21-d528e1d23603, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 0b1925c8-3ae5-4372-a3de-213551574fe1, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 663bd1f7-7a29-484c-bfce-d312646b1660, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: c6d325c3-cddf-4528-8ba6-a083f6511284, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External } } } }, OmicronZoneConfig { id: c30b7bc1-fd10-4942-8055-ada48d2e3957, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External } } } }, OmicronZoneConfig { id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
15952024-07-19T17:45:55.184ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
15962024-07-19T17:45:55.184ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15972024-07-19T17:45:55.208ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
15982024-07-19T17:45:55.208ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }
15992024-07-19T17:45:55.230ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
16002024-07-19T17:45:55.230ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }
16012024-07-19T17:45:55.250ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
16022024-07-19T17:45:55.250ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16032024-07-19T17:45:55.270ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
16042024-07-19T17:45:55.270ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16052024-07-19T17:45:55.289ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
16062024-07-19T17:45:55.289ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16072024-07-19T17:45:55.309ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
16082024-07-19T17:45:55.309ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16092024-07-19T17:45:55.328ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
16102024-07-19T17:45:55.328ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16112024-07-19T17:45:55.348ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
16122024-07-19T17:45:55.348ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
16132024-07-19T17:45:55.369ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
16142024-07-19T17:45:55.369ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
16152024-07-19T17:45:55.389ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
16162024-07-19T17:45:55.389ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
16172024-07-19T17:45:55.408ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
16182024-07-19T17:45:55.408ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }
16192024-07-19T17:45:55.427ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
16202024-07-19T17:45:55.427ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }
16212024-07-19T17:45:55.448ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
16222024-07-19T17:45:55.448ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }
16232024-07-19T17:45:55.469ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
16242024-07-19T17:45:55.469ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }
16252024-07-19T17:45:55.489ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
16262024-07-19T17:45:55.489ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }
16272024-07-19T17:45:55.509ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
16282024-07-19T17:45:55.510ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }
16292024-07-19T17:45:55.530ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
16302024-07-19T17:45:55.531ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }
16312024-07-19T17:45:55.550ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
16322024-07-19T17:45:55.550ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }
16332024-07-19T17:45:55.570ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
16342024-07-19T17:45:55.570ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }
16352024-07-19T17:45:55.589ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
16362024-07-19T17:45:55.589ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/omicron-zones.json file = common/src/ledger.rs:177
16372024-07-19T17:45:55.589ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/omicron-zones.json file = common/src/ledger.rs:177
16382024-07-19T17:45:55.590ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
16392024-07-19T17:45:55.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
16402024-07-19T17:45:55.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
16412024-07-19T17:45:55.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
16422024-07-19T17:45:55.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
16432024-07-19T17:45:55.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
16442024-07-19T17:45:55.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
16452024-07-19T17:45:55.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
16462024-07-19T17:45:55.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
16472024-07-19T17:45:55.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
16482024-07-19T17:45:55.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
16492024-07-19T17:45:55.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
16502024-07-19T17:45:55.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
16512024-07-19T17:45:55.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
16522024-07-19T17:45:55.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
16532024-07-19T17:45:55.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
16542024-07-19T17:45:55.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
16552024-07-19T17:45:55.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
16562024-07-19T17:45:55.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
16572024-07-19T17:45:55.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
16582024-07-19T17:45:55.688ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203", TimeNotSynchronized), ("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52", TimeNotSynchronized), ("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd", TimeNotSynchronized), ("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660", TimeNotSynchronized), ("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec", TimeNotSynchronized), ("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9", TimeNotSynchronized), ("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7", TimeNotSynchronized), ("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb", TimeNotSynchronized), ("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1", TimeNotSynchronized), ("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957", TimeNotSynchronized), ("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1", TimeNotSynchronized), ("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b", TimeNotSynchronized), ("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1", TimeNotSynchronized), ("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b", TimeNotSynchronized), ("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1", TimeNotSynchronized), ("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284", TimeNotSynchronized), ("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603", TimeNotSynchronized), ("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17", TimeNotSynchronized), ("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 504171 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:37752 req_id = 38706219-c761-4414-9ba6-87b319228af6 response_code = 500 uri = /omicron-zones
16592024-07-19T17:45:55.688ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "38706219-c761-4414-9ba6-87b319228af6", "content-length": "124", "date": "Fri, 19 Jul 2024 17:45:55 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "38706219-c761-4414-9ba6-87b319228af6" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 1.191411395s sled_address = [fd00:1122:3344:101::1]:12345
16602024-07-19T17:45:56.880ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 8ae14f97-bd01-42ad-87d5-23eae9194521, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: bc60810c-eece-4cbd-a0f8-d04271924b7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bf3fc707-b452-4d30-a92b-e56b4d4122ee, kind: Service { id: bc60810c-eece-4cbd-a0f8-d04271924b7c }, name: Name("ntp-bc60810c-eece-4cbd-a0f8-d04271924b7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 196])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: c168b27b-f467-4350-a348-4006ce85a213, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 15be01e5-1765-45f0-b521-da99b395bdb3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: d8a1310d-09c1-4844-91b7-509f93b30cfd, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 59731bd7-ffc9-493f-a33c-a1cf5ba2df30, kind: Service { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1 }, name: Name("external-dns-1eb90f9c-ba99-428e-885b-5fa2df7ae2f1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 32])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 09fdc6fd-8297-436e-903d-b18710599203, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 005b2740-84d5-473e-ae59-aa79fd2fe5b6, kind: Service { id: 09fdc6fd-8297-436e-903d-b18710599203 }, name: Name("external-dns-09fdc6fd-8297-436e-903d-b18710599203"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 6])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: c6edf68c-f558-4f51-9aed-6eb91fb3ad51, kind: Service { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52 }, name: Name("nexus-87d65a87-e1e5-43e4-9cba-b4afd5e9ac52"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 117])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6088410e-a8fc-4d33-b593-6385e863291f, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: e6898038-bf11-41c8-b971-6ff3eed6d38b, kind: Service { id: 6088410e-a8fc-4d33-b593-6385e863291f }, name: Name("nexus-6088410e-a8fc-4d33-b593-6385e863291f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 27])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 742e0330-77de-4723-a145-74751d41c73b, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: ffe6d41d-1349-42b2-a887-49d734eef422, kind: Service { id: 742e0330-77de-4723-a145-74751d41c73b }, name: Name("nexus-742e0330-77de-4723-a145-74751d41c73b"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 199, 184])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8d644923-5967-4351-a1ad-fe8a938f33e1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 740bd3e0-b30c-4033-968a-5451c8d5adcd, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 3286a4ca-fffd-4ed0-9c21-d528e1d23603, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 0b1925c8-3ae5-4372-a3de-213551574fe1, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 663bd1f7-7a29-484c-bfce-d312646b1660, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: c6d325c3-cddf-4528-8ba6-a083f6511284, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External } } } }, OmicronZoneConfig { id: c30b7bc1-fd10-4942-8055-ada48d2e3957, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External } } } }, OmicronZoneConfig { id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
16612024-07-19T17:45:56.880ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
16622024-07-19T17:45:56.880ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }
16632024-07-19T17:45:56.904ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
16642024-07-19T17:45:56.904ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }
16652024-07-19T17:45:56.924ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
16662024-07-19T17:45:56.924ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }
16672024-07-19T17:45:56.945ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
16682024-07-19T17:45:56.945ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16692024-07-19T17:45:56.964ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
16702024-07-19T17:45:56.964ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16712024-07-19T17:45:56.982ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
16722024-07-19T17:45:56.982ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16732024-07-19T17:45:57.002ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
16742024-07-19T17:45:57.002ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16752024-07-19T17:45:57.023ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
16762024-07-19T17:45:57.023ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16772024-07-19T17:45:57.045ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
16782024-07-19T17:45:57.045ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
16792024-07-19T17:45:57.065ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
16802024-07-19T17:45:57.065ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
16812024-07-19T17:45:57.087ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
16822024-07-19T17:45:57.087ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
16832024-07-19T17:45:57.108ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
16842024-07-19T17:45:57.109ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }
16852024-07-19T17:45:57.131ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
16862024-07-19T17:45:57.131ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }
16872024-07-19T17:45:57.151ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
16882024-07-19T17:45:57.151ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }
16892024-07-19T17:45:57.170ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
16902024-07-19T17:45:57.170ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }
16912024-07-19T17:45:57.191ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
16922024-07-19T17:45:57.191ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }
16932024-07-19T17:45:57.211ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
16942024-07-19T17:45:57.211ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }
16952024-07-19T17:45:57.234ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
16962024-07-19T17:45:57.235ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }
16972024-07-19T17:45:57.256ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
16982024-07-19T17:45:57.256ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }
16992024-07-19T17:45:57.277ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
17002024-07-19T17:45:57.277ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }
17012024-07-19T17:45:57.298ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
17022024-07-19T17:45:57.298ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/omicron-zones.json file = common/src/ledger.rs:177
17032024-07-19T17:45:57.298ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/omicron-zones.json file = common/src/ledger.rs:177
17042024-07-19T17:45:57.298ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
17052024-07-19T17:45:57.398ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
17062024-07-19T17:45:57.398ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
17072024-07-19T17:45:57.398ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
17082024-07-19T17:45:57.398ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
17092024-07-19T17:45:57.398ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
17102024-07-19T17:45:57.398ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
17112024-07-19T17:45:57.398ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
17122024-07-19T17:45:57.398ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
17132024-07-19T17:45:57.398ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
17142024-07-19T17:45:57.398ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
17152024-07-19T17:45:57.398ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
17162024-07-19T17:45:57.398ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
17172024-07-19T17:45:57.398ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
17182024-07-19T17:45:57.398ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
17192024-07-19T17:45:57.398ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
17202024-07-19T17:45:57.398ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
17212024-07-19T17:45:57.398ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
17222024-07-19T17:45:57.398ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
17232024-07-19T17:45:57.398ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
17242024-07-19T17:45:57.398ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957", TimeNotSynchronized), ("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1", TimeNotSynchronized), ("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1", TimeNotSynchronized), ("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b", TimeNotSynchronized), ("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17", TimeNotSynchronized), ("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52", TimeNotSynchronized), ("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9", TimeNotSynchronized), ("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284", TimeNotSynchronized), ("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b", TimeNotSynchronized), ("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660", TimeNotSynchronized), ("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203", TimeNotSynchronized), ("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f", TimeNotSynchronized), ("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603", TimeNotSynchronized), ("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1", TimeNotSynchronized), ("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec", TimeNotSynchronized), ("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd", TimeNotSynchronized), ("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1", TimeNotSynchronized), ("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7", TimeNotSynchronized), ("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 517928 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:37752 req_id = 943b5daa-f42e-4315-ba70-8dfeeb084b49 response_code = 500 uri = /omicron-zones
17252024-07-19T17:45:57.398ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "943b5daa-f42e-4315-ba70-8dfeeb084b49", "content-length": "124", "date": "Fri, 19 Jul 2024 17:45:56 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "943b5daa-f42e-4315-ba70-8dfeeb084b49" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 666.726847ms sled_address = [fd00:1122:3344:101::1]:12345
17262024-07-19T17:45:58.066ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 8ae14f97-bd01-42ad-87d5-23eae9194521, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: bc60810c-eece-4cbd-a0f8-d04271924b7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bf3fc707-b452-4d30-a92b-e56b4d4122ee, kind: Service { id: bc60810c-eece-4cbd-a0f8-d04271924b7c }, name: Name("ntp-bc60810c-eece-4cbd-a0f8-d04271924b7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 196])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: c168b27b-f467-4350-a348-4006ce85a213, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 15be01e5-1765-45f0-b521-da99b395bdb3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: d8a1310d-09c1-4844-91b7-509f93b30cfd, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 59731bd7-ffc9-493f-a33c-a1cf5ba2df30, kind: Service { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1 }, name: Name("external-dns-1eb90f9c-ba99-428e-885b-5fa2df7ae2f1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 32])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 09fdc6fd-8297-436e-903d-b18710599203, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 005b2740-84d5-473e-ae59-aa79fd2fe5b6, kind: Service { id: 09fdc6fd-8297-436e-903d-b18710599203 }, name: Name("external-dns-09fdc6fd-8297-436e-903d-b18710599203"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 6])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: c6edf68c-f558-4f51-9aed-6eb91fb3ad51, kind: Service { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52 }, name: Name("nexus-87d65a87-e1e5-43e4-9cba-b4afd5e9ac52"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 117])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6088410e-a8fc-4d33-b593-6385e863291f, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: e6898038-bf11-41c8-b971-6ff3eed6d38b, kind: Service { id: 6088410e-a8fc-4d33-b593-6385e863291f }, name: Name("nexus-6088410e-a8fc-4d33-b593-6385e863291f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 27])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 742e0330-77de-4723-a145-74751d41c73b, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: ffe6d41d-1349-42b2-a887-49d734eef422, kind: Service { id: 742e0330-77de-4723-a145-74751d41c73b }, name: Name("nexus-742e0330-77de-4723-a145-74751d41c73b"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 199, 184])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8d644923-5967-4351-a1ad-fe8a938f33e1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 740bd3e0-b30c-4033-968a-5451c8d5adcd, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 3286a4ca-fffd-4ed0-9c21-d528e1d23603, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 0b1925c8-3ae5-4372-a3de-213551574fe1, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 663bd1f7-7a29-484c-bfce-d312646b1660, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: c6d325c3-cddf-4528-8ba6-a083f6511284, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External } } } }, OmicronZoneConfig { id: c30b7bc1-fd10-4942-8055-ada48d2e3957, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External } } } }, OmicronZoneConfig { id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
17272024-07-19T17:45:58.067ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
17282024-07-19T17:45:58.067ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }
17292024-07-19T17:45:58.091ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
17302024-07-19T17:45:58.091ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }
17312024-07-19T17:45:58.112ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
17322024-07-19T17:45:58.112ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }
17332024-07-19T17:45:58.132ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
17342024-07-19T17:45:58.132ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17352024-07-19T17:45:58.153ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
17362024-07-19T17:45:58.153ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17372024-07-19T17:45:58.172ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
17382024-07-19T17:45:58.172ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17392024-07-19T17:45:58.190ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
17402024-07-19T17:45:58.190ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17412024-07-19T17:45:58.210ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
17422024-07-19T17:45:58.210ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17432024-07-19T17:45:58.230ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
17442024-07-19T17:45:58.230ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
17452024-07-19T17:45:58.249ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
17462024-07-19T17:45:58.249ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
17472024-07-19T17:45:58.269ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
17482024-07-19T17:45:58.269ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
17492024-07-19T17:45:58.290ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
17502024-07-19T17:45:58.290ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }
17512024-07-19T17:45:58.312ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
17522024-07-19T17:45:58.312ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }
17532024-07-19T17:45:58.332ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
17542024-07-19T17:45:58.332ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }
17552024-07-19T17:45:58.351ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
17562024-07-19T17:45:58.351ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }
17572024-07-19T17:45:58.371ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
17582024-07-19T17:45:58.371ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }
17592024-07-19T17:45:58.392ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
17602024-07-19T17:45:58.392ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }
17612024-07-19T17:45:58.413ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
17622024-07-19T17:45:58.413ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }
17632024-07-19T17:45:58.433ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
17642024-07-19T17:45:58.433ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }
17652024-07-19T17:45:58.454ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
17662024-07-19T17:45:58.454ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }
17672024-07-19T17:45:58.474ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
17682024-07-19T17:45:58.474ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/omicron-zones.json file = common/src/ledger.rs:177
17692024-07-19T17:45:58.475ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/omicron-zones.json file = common/src/ledger.rs:177
17702024-07-19T17:45:58.475ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
17712024-07-19T17:45:58.573ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
17722024-07-19T17:45:58.573ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
17732024-07-19T17:45:58.573ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
17742024-07-19T17:45:58.573ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
17752024-07-19T17:45:58.573ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
17762024-07-19T17:45:58.573ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
17772024-07-19T17:45:58.573ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
17782024-07-19T17:45:58.573ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
17792024-07-19T17:45:58.573ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
17802024-07-19T17:45:58.573ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
17812024-07-19T17:45:58.574ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
17822024-07-19T17:45:58.574ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
17832024-07-19T17:45:58.574ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
17842024-07-19T17:45:58.574ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
17852024-07-19T17:45:58.574ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
17862024-07-19T17:45:58.574ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
17872024-07-19T17:45:58.574ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
17882024-07-19T17:45:58.574ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
17892024-07-19T17:45:58.574ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
17902024-07-19T17:45:58.574ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec", TimeNotSynchronized), ("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1", TimeNotSynchronized), ("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb", TimeNotSynchronized), ("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957", TimeNotSynchronized), ("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17", TimeNotSynchronized), ("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284", TimeNotSynchronized), ("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b", TimeNotSynchronized), ("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203", TimeNotSynchronized), ("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1", TimeNotSynchronized), ("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660", TimeNotSynchronized), ("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52", TimeNotSynchronized), ("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9", TimeNotSynchronized), ("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1", TimeNotSynchronized), ("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603", TimeNotSynchronized), ("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7", TimeNotSynchronized), ("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1", TimeNotSynchronized), ("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd", TimeNotSynchronized), ("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f", TimeNotSynchronized), ("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 507419 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:37752 req_id = 335349a7-9402-43a6-a6de-63d3ad659fae response_code = 500 uri = /omicron-zones
17912024-07-19T17:45:58.574ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "335349a7-9402-43a6-a6de-63d3ad659fae", "content-length": "124", "date": "Fri, 19 Jul 2024 17:45:58 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "335349a7-9402-43a6-a6de-63d3ad659fae" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 1.527556121s sled_address = [fd00:1122:3344:101::1]:12345
17922024-07-19T17:45:58.941ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 2.283909868s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
17932024-07-19T17:46:00.102ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 8ae14f97-bd01-42ad-87d5-23eae9194521, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: bc60810c-eece-4cbd-a0f8-d04271924b7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bf3fc707-b452-4d30-a92b-e56b4d4122ee, kind: Service { id: bc60810c-eece-4cbd-a0f8-d04271924b7c }, name: Name("ntp-bc60810c-eece-4cbd-a0f8-d04271924b7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 196])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: c168b27b-f467-4350-a348-4006ce85a213, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 15be01e5-1765-45f0-b521-da99b395bdb3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: d8a1310d-09c1-4844-91b7-509f93b30cfd, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 59731bd7-ffc9-493f-a33c-a1cf5ba2df30, kind: Service { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1 }, name: Name("external-dns-1eb90f9c-ba99-428e-885b-5fa2df7ae2f1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 32])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 09fdc6fd-8297-436e-903d-b18710599203, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 005b2740-84d5-473e-ae59-aa79fd2fe5b6, kind: Service { id: 09fdc6fd-8297-436e-903d-b18710599203 }, name: Name("external-dns-09fdc6fd-8297-436e-903d-b18710599203"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 6])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: c6edf68c-f558-4f51-9aed-6eb91fb3ad51, kind: Service { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52 }, name: Name("nexus-87d65a87-e1e5-43e4-9cba-b4afd5e9ac52"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 117])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6088410e-a8fc-4d33-b593-6385e863291f, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: e6898038-bf11-41c8-b971-6ff3eed6d38b, kind: Service { id: 6088410e-a8fc-4d33-b593-6385e863291f }, name: Name("nexus-6088410e-a8fc-4d33-b593-6385e863291f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 27])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 742e0330-77de-4723-a145-74751d41c73b, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: ffe6d41d-1349-42b2-a887-49d734eef422, kind: Service { id: 742e0330-77de-4723-a145-74751d41c73b }, name: Name("nexus-742e0330-77de-4723-a145-74751d41c73b"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 199, 184])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8d644923-5967-4351-a1ad-fe8a938f33e1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 740bd3e0-b30c-4033-968a-5451c8d5adcd, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 3286a4ca-fffd-4ed0-9c21-d528e1d23603, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 0b1925c8-3ae5-4372-a3de-213551574fe1, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 663bd1f7-7a29-484c-bfce-d312646b1660, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: c6d325c3-cddf-4528-8ba6-a083f6511284, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External } } } }, OmicronZoneConfig { id: c30b7bc1-fd10-4942-8055-ada48d2e3957, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External } } } }, OmicronZoneConfig { id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
17942024-07-19T17:46:00.102ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
17952024-07-19T17:46:00.102ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }
17962024-07-19T17:46:00.125ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
17972024-07-19T17:46:00.125ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }
17982024-07-19T17:46:00.146ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
17992024-07-19T17:46:00.146ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }
18002024-07-19T17:46:00.166ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
18012024-07-19T17:46:00.166ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18022024-07-19T17:46:00.187ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
18032024-07-19T17:46:00.187ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18042024-07-19T17:46:00.207ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
18052024-07-19T17:46:00.207ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18062024-07-19T17:46:00.228ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
18072024-07-19T17:46:00.228ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18082024-07-19T17:46:00.247ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
18092024-07-19T17:46:00.247ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18102024-07-19T17:46:00.265ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
18112024-07-19T17:46:00.265ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
18122024-07-19T17:46:00.286ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
18132024-07-19T17:46:00.286ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
18142024-07-19T17:46:00.305ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
18152024-07-19T17:46:00.305ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
18162024-07-19T17:46:00.325ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
18172024-07-19T17:46:00.325ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }
18182024-07-19T17:46:00.346ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
18192024-07-19T17:46:00.346ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }
18202024-07-19T17:46:00.366ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
18212024-07-19T17:46:00.366ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }
18222024-07-19T17:46:00.386ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
18232024-07-19T17:46:00.386ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }
18242024-07-19T17:46:00.407ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
18252024-07-19T17:46:00.407ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }
18262024-07-19T17:46:00.428ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
18272024-07-19T17:46:00.428ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }
18282024-07-19T17:46:00.450ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
18292024-07-19T17:46:00.450ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }
18302024-07-19T17:46:00.469ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
18312024-07-19T17:46:00.469ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }
18322024-07-19T17:46:00.491ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
18332024-07-19T17:46:00.491ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }
18342024-07-19T17:46:00.511ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
18352024-07-19T17:46:00.511ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/omicron-zones.json file = common/src/ledger.rs:177
18362024-07-19T17:46:00.512ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/omicron-zones.json file = common/src/ledger.rs:177
18372024-07-19T17:46:00.512ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
18382024-07-19T17:46:00.610ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
18392024-07-19T17:46:00.610ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
18402024-07-19T17:46:00.610ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
18412024-07-19T17:46:00.610ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
18422024-07-19T17:46:00.610ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
18432024-07-19T17:46:00.610ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
18442024-07-19T17:46:00.610ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
18452024-07-19T17:46:00.610ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
18462024-07-19T17:46:00.610ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
18472024-07-19T17:46:00.610ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
18482024-07-19T17:46:00.610ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
18492024-07-19T17:46:00.610ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
18502024-07-19T17:46:00.610ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
18512024-07-19T17:46:00.610ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
18522024-07-19T17:46:00.610ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
18532024-07-19T17:46:00.610ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
18542024-07-19T17:46:00.610ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
18552024-07-19T17:46:00.610ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
18562024-07-19T17:46:00.610ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
18572024-07-19T17:46:00.611ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1", TimeNotSynchronized), ("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1", TimeNotSynchronized), ("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b", TimeNotSynchronized), ("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1", TimeNotSynchronized), ("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284", TimeNotSynchronized), ("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7", TimeNotSynchronized), ("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957", TimeNotSynchronized), ("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd", TimeNotSynchronized), ("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1", TimeNotSynchronized), ("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660", TimeNotSynchronized), ("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b", TimeNotSynchronized), ("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603", TimeNotSynchronized), ("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52", TimeNotSynchronized), ("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203", TimeNotSynchronized), ("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f", TimeNotSynchronized), ("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb", TimeNotSynchronized), ("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17", TimeNotSynchronized), ("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec", TimeNotSynchronized), ("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 508470 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:37752 req_id = 63ff7b13-f19f-42cc-bf9d-c0e1cc6fe09a response_code = 500 uri = /omicron-zones
18582024-07-19T17:46:00.611ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "63ff7b13-f19f-42cc-bf9d-c0e1cc6fe09a", "content-length": "124", "date": "Fri, 19 Jul 2024 17:46:00 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "63ff7b13-f19f-42cc-bf9d-c0e1cc6fe09a" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 1.227112834s sled_address = [fd00:1122:3344:101::1]:12345
18592024-07-19T17:46:01.840ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 8ae14f97-bd01-42ad-87d5-23eae9194521, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: bc60810c-eece-4cbd-a0f8-d04271924b7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bf3fc707-b452-4d30-a92b-e56b4d4122ee, kind: Service { id: bc60810c-eece-4cbd-a0f8-d04271924b7c }, name: Name("ntp-bc60810c-eece-4cbd-a0f8-d04271924b7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 196])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: c168b27b-f467-4350-a348-4006ce85a213, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 15be01e5-1765-45f0-b521-da99b395bdb3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: d8a1310d-09c1-4844-91b7-509f93b30cfd, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 59731bd7-ffc9-493f-a33c-a1cf5ba2df30, kind: Service { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1 }, name: Name("external-dns-1eb90f9c-ba99-428e-885b-5fa2df7ae2f1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 32])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 09fdc6fd-8297-436e-903d-b18710599203, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 005b2740-84d5-473e-ae59-aa79fd2fe5b6, kind: Service { id: 09fdc6fd-8297-436e-903d-b18710599203 }, name: Name("external-dns-09fdc6fd-8297-436e-903d-b18710599203"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 6])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: c6edf68c-f558-4f51-9aed-6eb91fb3ad51, kind: Service { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52 }, name: Name("nexus-87d65a87-e1e5-43e4-9cba-b4afd5e9ac52"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 117])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6088410e-a8fc-4d33-b593-6385e863291f, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: e6898038-bf11-41c8-b971-6ff3eed6d38b, kind: Service { id: 6088410e-a8fc-4d33-b593-6385e863291f }, name: Name("nexus-6088410e-a8fc-4d33-b593-6385e863291f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 27])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 742e0330-77de-4723-a145-74751d41c73b, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: ffe6d41d-1349-42b2-a887-49d734eef422, kind: Service { id: 742e0330-77de-4723-a145-74751d41c73b }, name: Name("nexus-742e0330-77de-4723-a145-74751d41c73b"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 199, 184])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8d644923-5967-4351-a1ad-fe8a938f33e1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 740bd3e0-b30c-4033-968a-5451c8d5adcd, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 3286a4ca-fffd-4ed0-9c21-d528e1d23603, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 0b1925c8-3ae5-4372-a3de-213551574fe1, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 663bd1f7-7a29-484c-bfce-d312646b1660, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: c6d325c3-cddf-4528-8ba6-a083f6511284, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External } } } }, OmicronZoneConfig { id: c30b7bc1-fd10-4942-8055-ada48d2e3957, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External } } } }, OmicronZoneConfig { id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
18602024-07-19T17:46:01.840ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
18612024-07-19T17:46:01.840ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }
18622024-07-19T17:46:01.863ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
18632024-07-19T17:46:01.863ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }
18642024-07-19T17:46:01.886ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
18652024-07-19T17:46:01.886ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }
18662024-07-19T17:46:01.905ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
18672024-07-19T17:46:01.905ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18682024-07-19T17:46:01.927ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
18692024-07-19T17:46:01.927ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18702024-07-19T17:46:01.951ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
18712024-07-19T17:46:01.951ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18722024-07-19T17:46:01.974ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
18732024-07-19T17:46:01.974ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18742024-07-19T17:46:01.994ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
18752024-07-19T17:46:01.994ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18762024-07-19T17:46:02.015ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
18772024-07-19T17:46:02.015ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
18782024-07-19T17:46:02.037ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
18792024-07-19T17:46:02.037ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
18802024-07-19T17:46:02.058ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
18812024-07-19T17:46:02.058ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
18822024-07-19T17:46:02.080ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
18832024-07-19T17:46:02.080ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }
18842024-07-19T17:46:02.100ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
18852024-07-19T17:46:02.100ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }
18862024-07-19T17:46:02.120ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
18872024-07-19T17:46:02.121ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }
18882024-07-19T17:46:02.141ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
18892024-07-19T17:46:02.141ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }
18902024-07-19T17:46:02.161ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
18912024-07-19T17:46:02.161ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }
18922024-07-19T17:46:02.181ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
18932024-07-19T17:46:02.181ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }
18942024-07-19T17:46:02.202ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
18952024-07-19T17:46:02.202ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }
18962024-07-19T17:46:02.222ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
18972024-07-19T17:46:02.222ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }
18982024-07-19T17:46:02.241ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
18992024-07-19T17:46:02.241ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }
19002024-07-19T17:46:02.262ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
19012024-07-19T17:46:02.262ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/omicron-zones.json file = common/src/ledger.rs:177
19022024-07-19T17:46:02.262ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/omicron-zones.json file = common/src/ledger.rs:177
19032024-07-19T17:46:02.262ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
19042024-07-19T17:46:02.365ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
19052024-07-19T17:46:02.365ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
19062024-07-19T17:46:02.365ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
19072024-07-19T17:46:02.365ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
19082024-07-19T17:46:02.365ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
19092024-07-19T17:46:02.365ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
19102024-07-19T17:46:02.365ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
19112024-07-19T17:46:02.365ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
19122024-07-19T17:46:02.365ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
19132024-07-19T17:46:02.365ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
19142024-07-19T17:46:02.365ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
19152024-07-19T17:46:02.365ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
19162024-07-19T17:46:02.365ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
19172024-07-19T17:46:02.365ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
19182024-07-19T17:46:02.365ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
19192024-07-19T17:46:02.365ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
19202024-07-19T17:46:02.366ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
19212024-07-19T17:46:02.366ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
19222024-07-19T17:46:02.366ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
19232024-07-19T17:46:02.366ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1", TimeNotSynchronized), ("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb", TimeNotSynchronized), ("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b", TimeNotSynchronized), ("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957", TimeNotSynchronized), ("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec", TimeNotSynchronized), ("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1", TimeNotSynchronized), ("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603", TimeNotSynchronized), ("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b", TimeNotSynchronized), ("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660", TimeNotSynchronized), ("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52", TimeNotSynchronized), ("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1", TimeNotSynchronized), ("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203", TimeNotSynchronized), ("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9", TimeNotSynchronized), ("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284", TimeNotSynchronized), ("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7", TimeNotSynchronized), ("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17", TimeNotSynchronized), ("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1", TimeNotSynchronized), ("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f", TimeNotSynchronized), ("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 525747 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:37752 req_id = a59f0591-4621-4c8e-90c6-30e4047d3c9a response_code = 500 uri = /omicron-zones
19242024-07-19T17:46:02.366ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "a59f0591-4621-4c8e-90c6-30e4047d3c9a", "content-length": "124", "date": "Fri, 19 Jul 2024 17:46:01 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "a59f0591-4621-4c8e-90c6-30e4047d3c9a" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 2.348055205s sled_address = [fd00:1122:3344:101::1]:12345
19252024-07-19T17:46:04.714ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 8ae14f97-bd01-42ad-87d5-23eae9194521, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: bc60810c-eece-4cbd-a0f8-d04271924b7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bf3fc707-b452-4d30-a92b-e56b4d4122ee, kind: Service { id: bc60810c-eece-4cbd-a0f8-d04271924b7c }, name: Name("ntp-bc60810c-eece-4cbd-a0f8-d04271924b7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 196])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: c168b27b-f467-4350-a348-4006ce85a213, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 15be01e5-1765-45f0-b521-da99b395bdb3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: d8a1310d-09c1-4844-91b7-509f93b30cfd, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 59731bd7-ffc9-493f-a33c-a1cf5ba2df30, kind: Service { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1 }, name: Name("external-dns-1eb90f9c-ba99-428e-885b-5fa2df7ae2f1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 32])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 09fdc6fd-8297-436e-903d-b18710599203, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 005b2740-84d5-473e-ae59-aa79fd2fe5b6, kind: Service { id: 09fdc6fd-8297-436e-903d-b18710599203 }, name: Name("external-dns-09fdc6fd-8297-436e-903d-b18710599203"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 6])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: c6edf68c-f558-4f51-9aed-6eb91fb3ad51, kind: Service { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52 }, name: Name("nexus-87d65a87-e1e5-43e4-9cba-b4afd5e9ac52"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 117])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6088410e-a8fc-4d33-b593-6385e863291f, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: e6898038-bf11-41c8-b971-6ff3eed6d38b, kind: Service { id: 6088410e-a8fc-4d33-b593-6385e863291f }, name: Name("nexus-6088410e-a8fc-4d33-b593-6385e863291f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 27])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 742e0330-77de-4723-a145-74751d41c73b, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: ffe6d41d-1349-42b2-a887-49d734eef422, kind: Service { id: 742e0330-77de-4723-a145-74751d41c73b }, name: Name("nexus-742e0330-77de-4723-a145-74751d41c73b"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 199, 184])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8d644923-5967-4351-a1ad-fe8a938f33e1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 740bd3e0-b30c-4033-968a-5451c8d5adcd, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 3286a4ca-fffd-4ed0-9c21-d528e1d23603, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 0b1925c8-3ae5-4372-a3de-213551574fe1, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 663bd1f7-7a29-484c-bfce-d312646b1660, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: c6d325c3-cddf-4528-8ba6-a083f6511284, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External } } } }, OmicronZoneConfig { id: c30b7bc1-fd10-4942-8055-ada48d2e3957, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External } } } }, OmicronZoneConfig { id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
19262024-07-19T17:46:04.714ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
19272024-07-19T17:46:04.714ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }
19282024-07-19T17:46:04.738ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
19292024-07-19T17:46:04.738ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }
19302024-07-19T17:46:04.757ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
19312024-07-19T17:46:04.757ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }
19322024-07-19T17:46:04.778ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
19332024-07-19T17:46:04.778ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19342024-07-19T17:46:04.799ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
19352024-07-19T17:46:04.799ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19362024-07-19T17:46:04.818ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
19372024-07-19T17:46:04.818ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19382024-07-19T17:46:04.838ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
19392024-07-19T17:46:04.838ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19402024-07-19T17:46:04.858ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
19412024-07-19T17:46:04.858ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19422024-07-19T17:46:04.877ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
19432024-07-19T17:46:04.878ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
19442024-07-19T17:46:04.896ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
19452024-07-19T17:46:04.897ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
19462024-07-19T17:46:04.916ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
19472024-07-19T17:46:04.916ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
19482024-07-19T17:46:04.934ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
19492024-07-19T17:46:04.934ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }
19502024-07-19T17:46:04.952ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
19512024-07-19T17:46:04.952ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }
19522024-07-19T17:46:04.972ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
19532024-07-19T17:46:04.972ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }
19542024-07-19T17:46:04.995ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
19552024-07-19T17:46:04.995ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }
19562024-07-19T17:46:05.017ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
19572024-07-19T17:46:05.017ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }
19582024-07-19T17:46:05.036ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
19592024-07-19T17:46:05.036ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }
19602024-07-19T17:46:05.055ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
19612024-07-19T17:46:05.055ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }
19622024-07-19T17:46:05.074ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
19632024-07-19T17:46:05.074ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }
19642024-07-19T17:46:05.094ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
19652024-07-19T17:46:05.094ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }
19662024-07-19T17:46:05.117ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
19672024-07-19T17:46:05.117ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/omicron-zones.json file = common/src/ledger.rs:177
19682024-07-19T17:46:05.117ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/omicron-zones.json file = common/src/ledger.rs:177
19692024-07-19T17:46:05.117ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
19702024-07-19T17:46:05.213ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
19712024-07-19T17:46:05.214ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
19722024-07-19T17:46:05.214ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
19732024-07-19T17:46:05.214ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
19742024-07-19T17:46:05.214ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
19752024-07-19T17:46:05.214ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
19762024-07-19T17:46:05.214ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
19772024-07-19T17:46:05.214ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
19782024-07-19T17:46:05.214ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
19792024-07-19T17:46:05.214ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
19802024-07-19T17:46:05.214ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
19812024-07-19T17:46:05.214ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
19822024-07-19T17:46:05.214ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
19832024-07-19T17:46:05.214ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
19842024-07-19T17:46:05.214ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
19852024-07-19T17:46:05.214ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
19862024-07-19T17:46:05.214ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
19872024-07-19T17:46:05.214ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
19882024-07-19T17:46:05.214ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
19892024-07-19T17:46:05.214ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9", TimeNotSynchronized), ("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b", TimeNotSynchronized), ("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52", TimeNotSynchronized), ("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284", TimeNotSynchronized), ("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1", TimeNotSynchronized), ("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb", TimeNotSynchronized), ("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603", TimeNotSynchronized), ("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec", TimeNotSynchronized), ("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7", TimeNotSynchronized), ("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1", TimeNotSynchronized), ("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203", TimeNotSynchronized), ("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f", TimeNotSynchronized), ("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd", TimeNotSynchronized), ("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1", TimeNotSynchronized), ("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957", TimeNotSynchronized), ("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660", TimeNotSynchronized), ("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17", TimeNotSynchronized), ("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b", TimeNotSynchronized), ("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 501134 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:37752 req_id = 9541333b-af85-434f-9d61-a73be126c9af response_code = 500 uri = /omicron-zones
19902024-07-19T17:46:05.214ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "9541333b-af85-434f-9d61-a73be126c9af", "content-length": "124", "date": "Fri, 19 Jul 2024 17:46:04 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "9541333b-af85-434f-9d61-a73be126c9af" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 2.128911756s sled_address = [fd00:1122:3344:101::1]:12345
19912024-07-19T17:46:07.338ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 8ae14f97-bd01-42ad-87d5-23eae9194521, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: bc60810c-eece-4cbd-a0f8-d04271924b7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bf3fc707-b452-4d30-a92b-e56b4d4122ee, kind: Service { id: bc60810c-eece-4cbd-a0f8-d04271924b7c }, name: Name("ntp-bc60810c-eece-4cbd-a0f8-d04271924b7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 196])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: c168b27b-f467-4350-a348-4006ce85a213, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 15be01e5-1765-45f0-b521-da99b395bdb3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: d8a1310d-09c1-4844-91b7-509f93b30cfd, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 59731bd7-ffc9-493f-a33c-a1cf5ba2df30, kind: Service { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1 }, name: Name("external-dns-1eb90f9c-ba99-428e-885b-5fa2df7ae2f1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 32])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 09fdc6fd-8297-436e-903d-b18710599203, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 005b2740-84d5-473e-ae59-aa79fd2fe5b6, kind: Service { id: 09fdc6fd-8297-436e-903d-b18710599203 }, name: Name("external-dns-09fdc6fd-8297-436e-903d-b18710599203"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 6])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: c6edf68c-f558-4f51-9aed-6eb91fb3ad51, kind: Service { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52 }, name: Name("nexus-87d65a87-e1e5-43e4-9cba-b4afd5e9ac52"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 117])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6088410e-a8fc-4d33-b593-6385e863291f, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: e6898038-bf11-41c8-b971-6ff3eed6d38b, kind: Service { id: 6088410e-a8fc-4d33-b593-6385e863291f }, name: Name("nexus-6088410e-a8fc-4d33-b593-6385e863291f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 27])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 742e0330-77de-4723-a145-74751d41c73b, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: ffe6d41d-1349-42b2-a887-49d734eef422, kind: Service { id: 742e0330-77de-4723-a145-74751d41c73b }, name: Name("nexus-742e0330-77de-4723-a145-74751d41c73b"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 199, 184])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8d644923-5967-4351-a1ad-fe8a938f33e1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 740bd3e0-b30c-4033-968a-5451c8d5adcd, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 3286a4ca-fffd-4ed0-9c21-d528e1d23603, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 0b1925c8-3ae5-4372-a3de-213551574fe1, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 663bd1f7-7a29-484c-bfce-d312646b1660, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: c6d325c3-cddf-4528-8ba6-a083f6511284, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External } } } }, OmicronZoneConfig { id: c30b7bc1-fd10-4942-8055-ada48d2e3957, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External } } } }, OmicronZoneConfig { id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
19922024-07-19T17:46:07.338ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
19932024-07-19T17:46:07.338ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }
19942024-07-19T17:46:07.361ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
19952024-07-19T17:46:07.361ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }
19962024-07-19T17:46:07.381ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
19972024-07-19T17:46:07.381ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }
19982024-07-19T17:46:07.401ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
19992024-07-19T17:46:07.401ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20002024-07-19T17:46:07.419ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
20012024-07-19T17:46:07.419ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20022024-07-19T17:46:07.441ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
20032024-07-19T17:46:07.441ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20042024-07-19T17:46:07.461ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
20052024-07-19T17:46:07.461ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20062024-07-19T17:46:07.480ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
20072024-07-19T17:46:07.480ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20082024-07-19T17:46:07.498ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
20092024-07-19T17:46:07.498ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
20102024-07-19T17:46:07.518ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
20112024-07-19T17:46:07.518ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
20122024-07-19T17:46:07.538ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
20132024-07-19T17:46:07.538ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
20142024-07-19T17:46:07.556ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
20152024-07-19T17:46:07.556ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }
20162024-07-19T17:46:07.577ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
20172024-07-19T17:46:07.577ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }
20182024-07-19T17:46:07.596ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
20192024-07-19T17:46:07.596ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }
20202024-07-19T17:46:07.615ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
20212024-07-19T17:46:07.615ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }
20222024-07-19T17:46:07.635ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
20232024-07-19T17:46:07.635ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }
20242024-07-19T17:46:07.654ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
20252024-07-19T17:46:07.654ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }
20262024-07-19T17:46:07.675ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
20272024-07-19T17:46:07.675ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }
20282024-07-19T17:46:07.696ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
20292024-07-19T17:46:07.696ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }
20302024-07-19T17:46:07.714ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
20312024-07-19T17:46:07.714ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }
20322024-07-19T17:46:07.735ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
20332024-07-19T17:46:07.735ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/omicron-zones.json file = common/src/ledger.rs:177
20342024-07-19T17:46:07.735ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/omicron-zones.json file = common/src/ledger.rs:177
20352024-07-19T17:46:07.735ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
20362024-07-19T17:46:07.836ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
20372024-07-19T17:46:07.836ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
20382024-07-19T17:46:07.836ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
20392024-07-19T17:46:07.836ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
20402024-07-19T17:46:07.836ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
20412024-07-19T17:46:07.836ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
20422024-07-19T17:46:07.836ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
20432024-07-19T17:46:07.836ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
20442024-07-19T17:46:07.836ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
20452024-07-19T17:46:07.836ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
20462024-07-19T17:46:07.836ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
20472024-07-19T17:46:07.836ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
20482024-07-19T17:46:07.836ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
20492024-07-19T17:46:07.836ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
20502024-07-19T17:46:07.836ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
20512024-07-19T17:46:07.836ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
20522024-07-19T17:46:07.836ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
20532024-07-19T17:46:07.836ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
20542024-07-19T17:46:07.836ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
20552024-07-19T17:46:07.836ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1", TimeNotSynchronized), ("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203", TimeNotSynchronized), ("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd", TimeNotSynchronized), ("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1", TimeNotSynchronized), ("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660", TimeNotSynchronized), ("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7", TimeNotSynchronized), ("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec", TimeNotSynchronized), ("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1", TimeNotSynchronized), ("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb", TimeNotSynchronized), ("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1", TimeNotSynchronized), ("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b", TimeNotSynchronized), ("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603", TimeNotSynchronized), ("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f", TimeNotSynchronized), ("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52", TimeNotSynchronized), ("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9", TimeNotSynchronized), ("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957", TimeNotSynchronized), ("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b", TimeNotSynchronized), ("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17", TimeNotSynchronized), ("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 499406 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:37752 req_id = 0582975f-8782-4d5a-ac39-6f52bbef6505 response_code = 500 uri = /omicron-zones
20562024-07-19T17:46:07.836ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "0582975f-8782-4d5a-ac39-6f52bbef6505", "content-length": "124", "date": "Fri, 19 Jul 2024 17:46:07 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "0582975f-8782-4d5a-ac39-6f52bbef6505" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 3.34857497s sled_address = [fd00:1122:3344:101::1]:12345
20572024-07-19T17:46:11.177ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 8ae14f97-bd01-42ad-87d5-23eae9194521, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: bc60810c-eece-4cbd-a0f8-d04271924b7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bf3fc707-b452-4d30-a92b-e56b4d4122ee, kind: Service { id: bc60810c-eece-4cbd-a0f8-d04271924b7c }, name: Name("ntp-bc60810c-eece-4cbd-a0f8-d04271924b7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 196])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: c168b27b-f467-4350-a348-4006ce85a213, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 15be01e5-1765-45f0-b521-da99b395bdb3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: d8a1310d-09c1-4844-91b7-509f93b30cfd, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 59731bd7-ffc9-493f-a33c-a1cf5ba2df30, kind: Service { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1 }, name: Name("external-dns-1eb90f9c-ba99-428e-885b-5fa2df7ae2f1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 32])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 09fdc6fd-8297-436e-903d-b18710599203, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 005b2740-84d5-473e-ae59-aa79fd2fe5b6, kind: Service { id: 09fdc6fd-8297-436e-903d-b18710599203 }, name: Name("external-dns-09fdc6fd-8297-436e-903d-b18710599203"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 6])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: c6edf68c-f558-4f51-9aed-6eb91fb3ad51, kind: Service { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52 }, name: Name("nexus-87d65a87-e1e5-43e4-9cba-b4afd5e9ac52"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 117])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6088410e-a8fc-4d33-b593-6385e863291f, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: e6898038-bf11-41c8-b971-6ff3eed6d38b, kind: Service { id: 6088410e-a8fc-4d33-b593-6385e863291f }, name: Name("nexus-6088410e-a8fc-4d33-b593-6385e863291f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 27])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 742e0330-77de-4723-a145-74751d41c73b, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: ffe6d41d-1349-42b2-a887-49d734eef422, kind: Service { id: 742e0330-77de-4723-a145-74751d41c73b }, name: Name("nexus-742e0330-77de-4723-a145-74751d41c73b"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 199, 184])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8d644923-5967-4351-a1ad-fe8a938f33e1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 740bd3e0-b30c-4033-968a-5451c8d5adcd, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 3286a4ca-fffd-4ed0-9c21-d528e1d23603, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 0b1925c8-3ae5-4372-a3de-213551574fe1, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 663bd1f7-7a29-484c-bfce-d312646b1660, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: c6d325c3-cddf-4528-8ba6-a083f6511284, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External } } } }, OmicronZoneConfig { id: c30b7bc1-fd10-4942-8055-ada48d2e3957, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External } } } }, OmicronZoneConfig { id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
20582024-07-19T17:46:11.177ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
20592024-07-19T17:46:11.178ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }
20602024-07-19T17:46:11.202ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
20612024-07-19T17:46:11.202ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }
20622024-07-19T17:46:11.223ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
20632024-07-19T17:46:11.223ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }
20642024-07-19T17:46:11.245ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
20652024-07-19T17:46:11.245ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20662024-07-19T17:46:11.265ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
20672024-07-19T17:46:11.265ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20682024-07-19T17:46:11.285ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
20692024-07-19T17:46:11.285ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20702024-07-19T17:46:11.305ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
20712024-07-19T17:46:11.305ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20722024-07-19T17:46:11.323ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
20732024-07-19T17:46:11.324ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20742024-07-19T17:46:11.343ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
20752024-07-19T17:46:11.343ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
20762024-07-19T17:46:11.363ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
20772024-07-19T17:46:11.363ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
20782024-07-19T17:46:11.383ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
20792024-07-19T17:46:11.383ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
20802024-07-19T17:46:11.404ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
20812024-07-19T17:46:11.404ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }
20822024-07-19T17:46:11.427ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
20832024-07-19T17:46:11.427ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }
20842024-07-19T17:46:11.447ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
20852024-07-19T17:46:11.447ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }
20862024-07-19T17:46:11.467ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
20872024-07-19T17:46:11.467ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }
20882024-07-19T17:46:11.487ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
20892024-07-19T17:46:11.487ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }
20902024-07-19T17:46:11.507ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
20912024-07-19T17:46:11.507ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }
20922024-07-19T17:46:11.526ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
20932024-07-19T17:46:11.526ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }
20942024-07-19T17:46:11.547ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
20952024-07-19T17:46:11.547ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }
20962024-07-19T17:46:11.568ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
20972024-07-19T17:46:11.568ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }
20982024-07-19T17:46:11.588ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
20992024-07-19T17:46:11.588ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/omicron-zones.json file = common/src/ledger.rs:177
21002024-07-19T17:46:11.588ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/omicron-zones.json file = common/src/ledger.rs:177
21012024-07-19T17:46:11.588ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
21022024-07-19T17:46:11.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
21032024-07-19T17:46:11.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
21042024-07-19T17:46:11.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
21052024-07-19T17:46:11.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
21062024-07-19T17:46:11.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
21072024-07-19T17:46:11.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
21082024-07-19T17:46:11.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
21092024-07-19T17:46:11.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
21102024-07-19T17:46:11.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
21112024-07-19T17:46:11.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
21122024-07-19T17:46:11.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
21132024-07-19T17:46:11.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
21142024-07-19T17:46:11.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
21152024-07-19T17:46:11.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
21162024-07-19T17:46:11.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
21172024-07-19T17:46:11.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
21182024-07-19T17:46:11.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
21192024-07-19T17:46:11.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
21202024-07-19T17:46:11.688ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
21212024-07-19T17:46:11.688ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603", TimeNotSynchronized), ("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284", TimeNotSynchronized), ("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec", TimeNotSynchronized), ("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb", TimeNotSynchronized), ("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957", TimeNotSynchronized), ("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1", TimeNotSynchronized), ("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1", TimeNotSynchronized), ("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b", TimeNotSynchronized), ("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9", TimeNotSynchronized), ("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17", TimeNotSynchronized), ("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b", TimeNotSynchronized), ("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52", TimeNotSynchronized), ("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f", TimeNotSynchronized), ("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1", TimeNotSynchronized), ("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1", TimeNotSynchronized), ("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203", TimeNotSynchronized), ("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd", TimeNotSynchronized), ("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660", TimeNotSynchronized), ("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 512423 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:37752 req_id = 6f19aae3-4b78-4362-afe9-a24c38b0690b response_code = 500 uri = /omicron-zones
21222024-07-19T17:46:11.689ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "6f19aae3-4b78-4362-afe9-a24c38b0690b", "content-length": "124", "date": "Fri, 19 Jul 2024 17:46:11 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "6f19aae3-4b78-4362-afe9-a24c38b0690b" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 3.056985196s sled_address = [fd00:1122:3344:101::1]:12345
21232024-07-19T17:46:14.012ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:35537
21242024-07-19T17:46:14.012ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 48 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:35537 req_id = 98bb37d2-6ac7-4652-9b97-58d7d816dc2f response_code = 200 uri = /baseboard
21252024-07-19T17:46:14.738ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 8ae14f97-bd01-42ad-87d5-23eae9194521, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: bc60810c-eece-4cbd-a0f8-d04271924b7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bf3fc707-b452-4d30-a92b-e56b4d4122ee, kind: Service { id: bc60810c-eece-4cbd-a0f8-d04271924b7c }, name: Name("ntp-bc60810c-eece-4cbd-a0f8-d04271924b7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 196])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: c168b27b-f467-4350-a348-4006ce85a213, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 15be01e5-1765-45f0-b521-da99b395bdb3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: d8a1310d-09c1-4844-91b7-509f93b30cfd, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 59731bd7-ffc9-493f-a33c-a1cf5ba2df30, kind: Service { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1 }, name: Name("external-dns-1eb90f9c-ba99-428e-885b-5fa2df7ae2f1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 32])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 09fdc6fd-8297-436e-903d-b18710599203, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 005b2740-84d5-473e-ae59-aa79fd2fe5b6, kind: Service { id: 09fdc6fd-8297-436e-903d-b18710599203 }, name: Name("external-dns-09fdc6fd-8297-436e-903d-b18710599203"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 6])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: c6edf68c-f558-4f51-9aed-6eb91fb3ad51, kind: Service { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52 }, name: Name("nexus-87d65a87-e1e5-43e4-9cba-b4afd5e9ac52"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 117])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6088410e-a8fc-4d33-b593-6385e863291f, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: e6898038-bf11-41c8-b971-6ff3eed6d38b, kind: Service { id: 6088410e-a8fc-4d33-b593-6385e863291f }, name: Name("nexus-6088410e-a8fc-4d33-b593-6385e863291f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 27])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 742e0330-77de-4723-a145-74751d41c73b, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: ffe6d41d-1349-42b2-a887-49d734eef422, kind: Service { id: 742e0330-77de-4723-a145-74751d41c73b }, name: Name("nexus-742e0330-77de-4723-a145-74751d41c73b"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 199, 184])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8d644923-5967-4351-a1ad-fe8a938f33e1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 740bd3e0-b30c-4033-968a-5451c8d5adcd, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 3286a4ca-fffd-4ed0-9c21-d528e1d23603, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 0b1925c8-3ae5-4372-a3de-213551574fe1, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 663bd1f7-7a29-484c-bfce-d312646b1660, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: c6d325c3-cddf-4528-8ba6-a083f6511284, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External } } } }, OmicronZoneConfig { id: c30b7bc1-fd10-4942-8055-ada48d2e3957, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External } } } }, OmicronZoneConfig { id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
21262024-07-19T17:46:14.738ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
21272024-07-19T17:46:14.738ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }
21282024-07-19T17:46:14.762ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
21292024-07-19T17:46:14.764ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }
21302024-07-19T17:46:14.782ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
21312024-07-19T17:46:14.782ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }
21322024-07-19T17:46:14.803ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
21332024-07-19T17:46:14.803ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
21342024-07-19T17:46:14.823ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
21352024-07-19T17:46:14.823ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
21362024-07-19T17:46:14.843ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
21372024-07-19T17:46:14.843ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
21382024-07-19T17:46:14.865ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
21392024-07-19T17:46:14.865ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
21402024-07-19T17:46:14.886ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
21412024-07-19T17:46:14.886ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
21422024-07-19T17:46:14.904ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
21432024-07-19T17:46:14.904ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
21442024-07-19T17:46:14.926ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
21452024-07-19T17:46:14.926ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
21462024-07-19T17:46:14.946ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
21472024-07-19T17:46:14.946ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
21482024-07-19T17:46:14.968ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
21492024-07-19T17:46:14.968ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }
21502024-07-19T17:46:14.990ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
21512024-07-19T17:46:14.990ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }
21522024-07-19T17:46:15.012ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
21532024-07-19T17:46:15.012ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }
21542024-07-19T17:46:15.033ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
21552024-07-19T17:46:15.033ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }
21562024-07-19T17:46:15.055ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
21572024-07-19T17:46:15.055ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }
21582024-07-19T17:46:15.077ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
21592024-07-19T17:46:15.077ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }
21602024-07-19T17:46:15.098ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
21612024-07-19T17:46:15.098ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }
21622024-07-19T17:46:15.120ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
21632024-07-19T17:46:15.120ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }
21642024-07-19T17:46:15.141ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
21652024-07-19T17:46:15.141ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }
21662024-07-19T17:46:15.162ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
21672024-07-19T17:46:15.162ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/omicron-zones.json file = common/src/ledger.rs:177
21682024-07-19T17:46:15.162ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/omicron-zones.json file = common/src/ledger.rs:177
21692024-07-19T17:46:15.162ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
21702024-07-19T17:46:15.259ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
21712024-07-19T17:46:15.259ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
21722024-07-19T17:46:15.259ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
21732024-07-19T17:46:15.259ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
21742024-07-19T17:46:15.259ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
21752024-07-19T17:46:15.260ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
21762024-07-19T17:46:15.260ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
21772024-07-19T17:46:15.260ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
21782024-07-19T17:46:15.260ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
21792024-07-19T17:46:15.260ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
21802024-07-19T17:46:15.260ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
21812024-07-19T17:46:15.260ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
21822024-07-19T17:46:15.260ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
21832024-07-19T17:46:15.260ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
21842024-07-19T17:46:15.260ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
21852024-07-19T17:46:15.260ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
21862024-07-19T17:46:15.260ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
21872024-07-19T17:46:15.260ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
21882024-07-19T17:46:15.260ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
21892024-07-19T17:46:15.260ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1", TimeNotSynchronized), ("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f", TimeNotSynchronized), ("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603", TimeNotSynchronized), ("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1", TimeNotSynchronized), ("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd", TimeNotSynchronized), ("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec", TimeNotSynchronized), ("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9", TimeNotSynchronized), ("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7", TimeNotSynchronized), ("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb", TimeNotSynchronized), ("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b", TimeNotSynchronized), ("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17", TimeNotSynchronized), ("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1", TimeNotSynchronized), ("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203", TimeNotSynchronized), ("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b", TimeNotSynchronized), ("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660", TimeNotSynchronized), ("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52", TimeNotSynchronized), ("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284", TimeNotSynchronized), ("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957", TimeNotSynchronized), ("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 523049 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:37752 req_id = 452e37c0-a3b1-4c89-8620-5b4352c253d5 response_code = 500 uri = /omicron-zones
21902024-07-19T17:46:15.260ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "452e37c0-a3b1-4c89-8620-5b4352c253d5", "content-length": "124", "date": "Fri, 19 Jul 2024 17:46:14 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "452e37c0-a3b1-4c89-8620-5b4352c253d5" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 5.489105675s sled_address = [fd00:1122:3344:101::1]:12345
21912024-07-19T17:46:16.230ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 4.094255488s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
21922024-07-19T17:46:20.735ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 8ae14f97-bd01-42ad-87d5-23eae9194521, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: bc60810c-eece-4cbd-a0f8-d04271924b7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bf3fc707-b452-4d30-a92b-e56b4d4122ee, kind: Service { id: bc60810c-eece-4cbd-a0f8-d04271924b7c }, name: Name("ntp-bc60810c-eece-4cbd-a0f8-d04271924b7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 196])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: c168b27b-f467-4350-a348-4006ce85a213, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 15be01e5-1765-45f0-b521-da99b395bdb3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: d8a1310d-09c1-4844-91b7-509f93b30cfd, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 59731bd7-ffc9-493f-a33c-a1cf5ba2df30, kind: Service { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1 }, name: Name("external-dns-1eb90f9c-ba99-428e-885b-5fa2df7ae2f1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 32])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 09fdc6fd-8297-436e-903d-b18710599203, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 005b2740-84d5-473e-ae59-aa79fd2fe5b6, kind: Service { id: 09fdc6fd-8297-436e-903d-b18710599203 }, name: Name("external-dns-09fdc6fd-8297-436e-903d-b18710599203"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 6])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: c6edf68c-f558-4f51-9aed-6eb91fb3ad51, kind: Service { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52 }, name: Name("nexus-87d65a87-e1e5-43e4-9cba-b4afd5e9ac52"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 117])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6088410e-a8fc-4d33-b593-6385e863291f, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: e6898038-bf11-41c8-b971-6ff3eed6d38b, kind: Service { id: 6088410e-a8fc-4d33-b593-6385e863291f }, name: Name("nexus-6088410e-a8fc-4d33-b593-6385e863291f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 27])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 742e0330-77de-4723-a145-74751d41c73b, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: ffe6d41d-1349-42b2-a887-49d734eef422, kind: Service { id: 742e0330-77de-4723-a145-74751d41c73b }, name: Name("nexus-742e0330-77de-4723-a145-74751d41c73b"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 199, 184])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8d644923-5967-4351-a1ad-fe8a938f33e1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 740bd3e0-b30c-4033-968a-5451c8d5adcd, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 3286a4ca-fffd-4ed0-9c21-d528e1d23603, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 0b1925c8-3ae5-4372-a3de-213551574fe1, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 663bd1f7-7a29-484c-bfce-d312646b1660, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: c6d325c3-cddf-4528-8ba6-a083f6511284, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External } } } }, OmicronZoneConfig { id: c30b7bc1-fd10-4942-8055-ada48d2e3957, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External } } } }, OmicronZoneConfig { id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
21932024-07-19T17:46:20.736ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
21942024-07-19T17:46:20.736ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }
21952024-07-19T17:46:20.758ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
21962024-07-19T17:46:20.758ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }
21972024-07-19T17:46:20.778ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
21982024-07-19T17:46:20.778ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }
21992024-07-19T17:46:20.797ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
22002024-07-19T17:46:20.797ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22012024-07-19T17:46:20.816ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
22022024-07-19T17:46:20.816ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22032024-07-19T17:46:20.834ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
22042024-07-19T17:46:20.834ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22052024-07-19T17:46:20.853ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
22062024-07-19T17:46:20.853ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22072024-07-19T17:46:20.871ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
22082024-07-19T17:46:20.871ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22092024-07-19T17:46:20.889ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
22102024-07-19T17:46:20.889ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
22112024-07-19T17:46:20.908ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
22122024-07-19T17:46:20.908ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
22132024-07-19T17:46:20.927ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
22142024-07-19T17:46:20.927ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
22152024-07-19T17:46:20.946ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
22162024-07-19T17:46:20.946ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }
22172024-07-19T17:46:20.964ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
22182024-07-19T17:46:20.964ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }
22192024-07-19T17:46:20.983ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
22202024-07-19T17:46:20.983ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }
22212024-07-19T17:46:21.002ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
22222024-07-19T17:46:21.002ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }
22232024-07-19T17:46:21.021ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
22242024-07-19T17:46:21.021ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }
22252024-07-19T17:46:21.040ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
22262024-07-19T17:46:21.040ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }
22272024-07-19T17:46:21.059ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
22282024-07-19T17:46:21.059ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }
22292024-07-19T17:46:21.078ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
22302024-07-19T17:46:21.079ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }
22312024-07-19T17:46:21.097ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
22322024-07-19T17:46:21.097ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }
22332024-07-19T17:46:21.116ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
22342024-07-19T17:46:21.116ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/omicron-zones.json file = common/src/ledger.rs:177
22352024-07-19T17:46:21.117ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/omicron-zones.json file = common/src/ledger.rs:177
22362024-07-19T17:46:21.117ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
22372024-07-19T17:46:21.210ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
22382024-07-19T17:46:21.210ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
22392024-07-19T17:46:21.210ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
22402024-07-19T17:46:21.210ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
22412024-07-19T17:46:21.210ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
22422024-07-19T17:46:21.210ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
22432024-07-19T17:46:21.210ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
22442024-07-19T17:46:21.210ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
22452024-07-19T17:46:21.210ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
22462024-07-19T17:46:21.210ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
22472024-07-19T17:46:21.210ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
22482024-07-19T17:46:21.210ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
22492024-07-19T17:46:21.210ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
22502024-07-19T17:46:21.210ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
22512024-07-19T17:46:21.210ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
22522024-07-19T17:46:21.210ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
22532024-07-19T17:46:21.210ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
22542024-07-19T17:46:21.210ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
22552024-07-19T17:46:21.210ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
22562024-07-19T17:46:21.210ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd", TimeNotSynchronized), ("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1", TimeNotSynchronized), ("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f", TimeNotSynchronized), ("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284", TimeNotSynchronized), ("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7", TimeNotSynchronized), ("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603", TimeNotSynchronized), ("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb", TimeNotSynchronized), ("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17", TimeNotSynchronized), ("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52", TimeNotSynchronized), ("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203", TimeNotSynchronized), ("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1", TimeNotSynchronized), ("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b", TimeNotSynchronized), ("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660", TimeNotSynchronized), ("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9", TimeNotSynchronized), ("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec", TimeNotSynchronized), ("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957", TimeNotSynchronized), ("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1", TimeNotSynchronized), ("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b", TimeNotSynchronized), ("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 476360 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:37752 req_id = eb3befad-d4be-4005-ae7a-b699610e2329 response_code = 500 uri = /omicron-zones
22572024-07-19T17:46:21.211ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "eb3befad-d4be-4005-ae7a-b699610e2329", "content-length": "124", "date": "Fri, 19 Jul 2024 17:46:20 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "eb3befad-d4be-4005-ae7a-b699610e2329" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 3.173396656s sled_address = [fd00:1122:3344:101::1]:12345
22582024-07-19T17:46:24.377ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 8ae14f97-bd01-42ad-87d5-23eae9194521, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: bc60810c-eece-4cbd-a0f8-d04271924b7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bf3fc707-b452-4d30-a92b-e56b4d4122ee, kind: Service { id: bc60810c-eece-4cbd-a0f8-d04271924b7c }, name: Name("ntp-bc60810c-eece-4cbd-a0f8-d04271924b7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 196])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: c168b27b-f467-4350-a348-4006ce85a213, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 15be01e5-1765-45f0-b521-da99b395bdb3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: d8a1310d-09c1-4844-91b7-509f93b30cfd, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 59731bd7-ffc9-493f-a33c-a1cf5ba2df30, kind: Service { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1 }, name: Name("external-dns-1eb90f9c-ba99-428e-885b-5fa2df7ae2f1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 32])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 09fdc6fd-8297-436e-903d-b18710599203, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 005b2740-84d5-473e-ae59-aa79fd2fe5b6, kind: Service { id: 09fdc6fd-8297-436e-903d-b18710599203 }, name: Name("external-dns-09fdc6fd-8297-436e-903d-b18710599203"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 6])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: c6edf68c-f558-4f51-9aed-6eb91fb3ad51, kind: Service { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52 }, name: Name("nexus-87d65a87-e1e5-43e4-9cba-b4afd5e9ac52"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 117])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6088410e-a8fc-4d33-b593-6385e863291f, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: e6898038-bf11-41c8-b971-6ff3eed6d38b, kind: Service { id: 6088410e-a8fc-4d33-b593-6385e863291f }, name: Name("nexus-6088410e-a8fc-4d33-b593-6385e863291f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 27])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 742e0330-77de-4723-a145-74751d41c73b, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: ffe6d41d-1349-42b2-a887-49d734eef422, kind: Service { id: 742e0330-77de-4723-a145-74751d41c73b }, name: Name("nexus-742e0330-77de-4723-a145-74751d41c73b"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 199, 184])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8d644923-5967-4351-a1ad-fe8a938f33e1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 740bd3e0-b30c-4033-968a-5451c8d5adcd, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 3286a4ca-fffd-4ed0-9c21-d528e1d23603, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 0b1925c8-3ae5-4372-a3de-213551574fe1, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 663bd1f7-7a29-484c-bfce-d312646b1660, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: c6d325c3-cddf-4528-8ba6-a083f6511284, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External } } } }, OmicronZoneConfig { id: c30b7bc1-fd10-4942-8055-ada48d2e3957, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External } } } }, OmicronZoneConfig { id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
22592024-07-19T17:46:24.378ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
22602024-07-19T17:46:24.378ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }
22612024-07-19T17:46:24.401ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
22622024-07-19T17:46:24.401ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }
22632024-07-19T17:46:24.424ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
22642024-07-19T17:46:24.424ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }
22652024-07-19T17:46:24.444ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
22662024-07-19T17:46:24.444ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22672024-07-19T17:46:24.464ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
22682024-07-19T17:46:24.464ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22692024-07-19T17:46:24.485ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
22702024-07-19T17:46:24.485ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22712024-07-19T17:46:24.505ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
22722024-07-19T17:46:24.505ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22732024-07-19T17:46:24.526ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
22742024-07-19T17:46:24.526ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22752024-07-19T17:46:24.547ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
22762024-07-19T17:46:24.547ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
22772024-07-19T17:46:24.568ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
22782024-07-19T17:46:24.568ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
22792024-07-19T17:46:24.590ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
22802024-07-19T17:46:24.590ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
22812024-07-19T17:46:24.611ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
22822024-07-19T17:46:24.611ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }
22832024-07-19T17:46:24.631ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
22842024-07-19T17:46:24.631ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }
22852024-07-19T17:46:24.653ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
22862024-07-19T17:46:24.653ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }
22872024-07-19T17:46:24.673ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
22882024-07-19T17:46:24.673ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }
22892024-07-19T17:46:24.693ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
22902024-07-19T17:46:24.693ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }
22912024-07-19T17:46:24.714ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
22922024-07-19T17:46:24.714ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }
22932024-07-19T17:46:24.735ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
22942024-07-19T17:46:24.735ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }
22952024-07-19T17:46:24.754ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
22962024-07-19T17:46:24.754ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }
22972024-07-19T17:46:24.776ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
22982024-07-19T17:46:24.776ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }
22992024-07-19T17:46:24.799ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
23002024-07-19T17:46:24.799ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/omicron-zones.json file = common/src/ledger.rs:177
23012024-07-19T17:46:24.800ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/omicron-zones.json file = common/src/ledger.rs:177
23022024-07-19T17:46:24.800ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
23032024-07-19T17:46:24.899ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
23042024-07-19T17:46:24.899ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
23052024-07-19T17:46:24.899ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
23062024-07-19T17:46:24.899ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
23072024-07-19T17:46:24.899ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
23082024-07-19T17:46:24.899ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
23092024-07-19T17:46:24.899ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
23102024-07-19T17:46:24.899ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
23112024-07-19T17:46:24.899ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
23122024-07-19T17:46:24.899ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
23132024-07-19T17:46:24.899ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
23142024-07-19T17:46:24.899ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
23152024-07-19T17:46:24.899ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
23162024-07-19T17:46:24.899ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
23172024-07-19T17:46:24.899ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
23182024-07-19T17:46:24.899ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
23192024-07-19T17:46:24.899ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
23202024-07-19T17:46:24.899ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
23212024-07-19T17:46:24.899ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
23222024-07-19T17:46:24.899ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f", TimeNotSynchronized), ("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603", TimeNotSynchronized), ("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203", TimeNotSynchronized), ("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b", TimeNotSynchronized), ("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd", TimeNotSynchronized), ("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284", TimeNotSynchronized), ("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7", TimeNotSynchronized), ("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb", TimeNotSynchronized), ("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957", TimeNotSynchronized), ("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17", TimeNotSynchronized), ("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1", TimeNotSynchronized), ("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec", TimeNotSynchronized), ("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1", TimeNotSynchronized), ("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1", TimeNotSynchronized), ("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b", TimeNotSynchronized), ("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660", TimeNotSynchronized), ("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9", TimeNotSynchronized), ("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52", TimeNotSynchronized), ("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 523241 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:37752 req_id = 2c7056af-12ff-46fb-8d66-9fb9996b8e8e response_code = 500 uri = /omicron-zones
23232024-07-19T17:46:24.900ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "2c7056af-12ff-46fb-8d66-9fb9996b8e8e", "content-length": "124", "date": "Fri, 19 Jul 2024 17:46:24 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "2c7056af-12ff-46fb-8d66-9fb9996b8e8e" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 7.799738611s sled_address = [fd00:1122:3344:101::1]:12345
23242024-07-19T17:46:32.680ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 8ae14f97-bd01-42ad-87d5-23eae9194521, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: bc60810c-eece-4cbd-a0f8-d04271924b7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bf3fc707-b452-4d30-a92b-e56b4d4122ee, kind: Service { id: bc60810c-eece-4cbd-a0f8-d04271924b7c }, name: Name("ntp-bc60810c-eece-4cbd-a0f8-d04271924b7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 196])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: c168b27b-f467-4350-a348-4006ce85a213, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 15be01e5-1765-45f0-b521-da99b395bdb3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: d8a1310d-09c1-4844-91b7-509f93b30cfd, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 59731bd7-ffc9-493f-a33c-a1cf5ba2df30, kind: Service { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1 }, name: Name("external-dns-1eb90f9c-ba99-428e-885b-5fa2df7ae2f1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 32])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 09fdc6fd-8297-436e-903d-b18710599203, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 005b2740-84d5-473e-ae59-aa79fd2fe5b6, kind: Service { id: 09fdc6fd-8297-436e-903d-b18710599203 }, name: Name("external-dns-09fdc6fd-8297-436e-903d-b18710599203"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 6])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: c6edf68c-f558-4f51-9aed-6eb91fb3ad51, kind: Service { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52 }, name: Name("nexus-87d65a87-e1e5-43e4-9cba-b4afd5e9ac52"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 117])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6088410e-a8fc-4d33-b593-6385e863291f, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: e6898038-bf11-41c8-b971-6ff3eed6d38b, kind: Service { id: 6088410e-a8fc-4d33-b593-6385e863291f }, name: Name("nexus-6088410e-a8fc-4d33-b593-6385e863291f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 27])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 742e0330-77de-4723-a145-74751d41c73b, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: ffe6d41d-1349-42b2-a887-49d734eef422, kind: Service { id: 742e0330-77de-4723-a145-74751d41c73b }, name: Name("nexus-742e0330-77de-4723-a145-74751d41c73b"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 199, 184])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8d644923-5967-4351-a1ad-fe8a938f33e1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 740bd3e0-b30c-4033-968a-5451c8d5adcd, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 3286a4ca-fffd-4ed0-9c21-d528e1d23603, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 0b1925c8-3ae5-4372-a3de-213551574fe1, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 663bd1f7-7a29-484c-bfce-d312646b1660, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: c6d325c3-cddf-4528-8ba6-a083f6511284, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External } } } }, OmicronZoneConfig { id: c30b7bc1-fd10-4942-8055-ada48d2e3957, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External } } } }, OmicronZoneConfig { id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
23252024-07-19T17:46:32.680ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
23262024-07-19T17:46:32.680ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }
23272024-07-19T17:46:32.703ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
23282024-07-19T17:46:32.703ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }
23292024-07-19T17:46:32.723ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
23302024-07-19T17:46:32.723ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }
23312024-07-19T17:46:32.741ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
23322024-07-19T17:46:32.741ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
23332024-07-19T17:46:32.759ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
23342024-07-19T17:46:32.759ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
23352024-07-19T17:46:32.779ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
23362024-07-19T17:46:32.779ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
23372024-07-19T17:46:32.797ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
23382024-07-19T17:46:32.797ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
23392024-07-19T17:46:32.817ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
23402024-07-19T17:46:32.817ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
23412024-07-19T17:46:32.834ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
23422024-07-19T17:46:32.835ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
23432024-07-19T17:46:32.854ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
23442024-07-19T17:46:32.854ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
23452024-07-19T17:46:32.876ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
23462024-07-19T17:46:32.876ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
23472024-07-19T17:46:32.895ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
23482024-07-19T17:46:32.895ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }
23492024-07-19T17:46:32.914ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
23502024-07-19T17:46:32.914ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }
23512024-07-19T17:46:32.932ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
23522024-07-19T17:46:32.932ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }
23532024-07-19T17:46:32.951ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
23542024-07-19T17:46:32.951ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }
23552024-07-19T17:46:32.970ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
23562024-07-19T17:46:32.970ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }
23572024-07-19T17:46:32.988ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
23582024-07-19T17:46:32.988ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }
23592024-07-19T17:46:33.007ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
23602024-07-19T17:46:33.007ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }
23612024-07-19T17:46:33.026ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
23622024-07-19T17:46:33.026ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }
23632024-07-19T17:46:33.045ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
23642024-07-19T17:46:33.045ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }
23652024-07-19T17:46:33.063ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
23662024-07-19T17:46:33.063ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/omicron-zones.json file = common/src/ledger.rs:177
23672024-07-19T17:46:33.064ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/omicron-zones.json file = common/src/ledger.rs:177
23682024-07-19T17:46:33.064ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
23692024-07-19T17:46:33.155ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
23702024-07-19T17:46:33.155ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
23712024-07-19T17:46:33.155ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
23722024-07-19T17:46:33.155ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
23732024-07-19T17:46:33.155ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
23742024-07-19T17:46:33.155ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
23752024-07-19T17:46:33.155ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
23762024-07-19T17:46:33.155ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
23772024-07-19T17:46:33.155ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
23782024-07-19T17:46:33.155ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
23792024-07-19T17:46:33.155ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
23802024-07-19T17:46:33.155ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
23812024-07-19T17:46:33.155ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
23822024-07-19T17:46:33.155ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
23832024-07-19T17:46:33.155ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
23842024-07-19T17:46:33.155ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
23852024-07-19T17:46:33.155ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
23862024-07-19T17:46:33.155ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
23872024-07-19T17:46:33.155ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
23882024-07-19T17:46:33.155ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b", TimeNotSynchronized), ("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1", TimeNotSynchronized), ("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660", TimeNotSynchronized), ("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284", TimeNotSynchronized), ("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb", TimeNotSynchronized), ("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957", TimeNotSynchronized), ("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17", TimeNotSynchronized), ("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603", TimeNotSynchronized), ("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7", TimeNotSynchronized), ("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b", TimeNotSynchronized), ("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1", TimeNotSynchronized), ("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd", TimeNotSynchronized), ("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9", TimeNotSynchronized), ("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203", TimeNotSynchronized), ("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52", TimeNotSynchronized), ("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec", TimeNotSynchronized), ("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1", TimeNotSynchronized), ("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f", TimeNotSynchronized), ("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 476587 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:37752 req_id = 4b65e5cd-722e-4afe-b0dc-30db340459c3 response_code = 500 uri = /omicron-zones
23892024-07-19T17:46:33.155ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "4b65e5cd-722e-4afe-b0dc-30db340459c3", "content-length": "124", "date": "Fri, 19 Jul 2024 17:46:32 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "4b65e5cd-722e-4afe-b0dc-30db340459c3" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 3.685750344s sled_address = [fd00:1122:3344:101::1]:12345
23902024-07-19T17:46:35.308ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 8.142177937s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
23912024-07-19T17:46:36.832ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 8ae14f97-bd01-42ad-87d5-23eae9194521, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: bc60810c-eece-4cbd-a0f8-d04271924b7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bf3fc707-b452-4d30-a92b-e56b4d4122ee, kind: Service { id: bc60810c-eece-4cbd-a0f8-d04271924b7c }, name: Name("ntp-bc60810c-eece-4cbd-a0f8-d04271924b7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 196])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: c168b27b-f467-4350-a348-4006ce85a213, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 15be01e5-1765-45f0-b521-da99b395bdb3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: d8a1310d-09c1-4844-91b7-509f93b30cfd, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 59731bd7-ffc9-493f-a33c-a1cf5ba2df30, kind: Service { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1 }, name: Name("external-dns-1eb90f9c-ba99-428e-885b-5fa2df7ae2f1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 32])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 09fdc6fd-8297-436e-903d-b18710599203, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 005b2740-84d5-473e-ae59-aa79fd2fe5b6, kind: Service { id: 09fdc6fd-8297-436e-903d-b18710599203 }, name: Name("external-dns-09fdc6fd-8297-436e-903d-b18710599203"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 6])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: c6edf68c-f558-4f51-9aed-6eb91fb3ad51, kind: Service { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52 }, name: Name("nexus-87d65a87-e1e5-43e4-9cba-b4afd5e9ac52"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 117])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6088410e-a8fc-4d33-b593-6385e863291f, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: e6898038-bf11-41c8-b971-6ff3eed6d38b, kind: Service { id: 6088410e-a8fc-4d33-b593-6385e863291f }, name: Name("nexus-6088410e-a8fc-4d33-b593-6385e863291f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 27])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 742e0330-77de-4723-a145-74751d41c73b, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: ffe6d41d-1349-42b2-a887-49d734eef422, kind: Service { id: 742e0330-77de-4723-a145-74751d41c73b }, name: Name("nexus-742e0330-77de-4723-a145-74751d41c73b"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 199, 184])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8d644923-5967-4351-a1ad-fe8a938f33e1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 740bd3e0-b30c-4033-968a-5451c8d5adcd, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 3286a4ca-fffd-4ed0-9c21-d528e1d23603, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 0b1925c8-3ae5-4372-a3de-213551574fe1, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 663bd1f7-7a29-484c-bfce-d312646b1660, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: c6d325c3-cddf-4528-8ba6-a083f6511284, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External } } } }, OmicronZoneConfig { id: c30b7bc1-fd10-4942-8055-ada48d2e3957, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External } } } }, OmicronZoneConfig { id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
23922024-07-19T17:46:36.833ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
23932024-07-19T17:46:36.833ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }
23942024-07-19T17:46:36.856ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
23952024-07-19T17:46:36.856ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }
23962024-07-19T17:46:36.877ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
23972024-07-19T17:46:36.877ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }
23982024-07-19T17:46:36.898ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
23992024-07-19T17:46:36.898ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24002024-07-19T17:46:36.917ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
24012024-07-19T17:46:36.917ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24022024-07-19T17:46:36.935ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
24032024-07-19T17:46:36.935ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24042024-07-19T17:46:36.953ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
24052024-07-19T17:46:36.953ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24062024-07-19T17:46:36.972ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
24072024-07-19T17:46:36.972ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24082024-07-19T17:46:36.990ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
24092024-07-19T17:46:36.990ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
24102024-07-19T17:46:37.008ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
24112024-07-19T17:46:37.008ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
24122024-07-19T17:46:37.027ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
24132024-07-19T17:46:37.027ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
24142024-07-19T17:46:37.045ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
24152024-07-19T17:46:37.045ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }
24162024-07-19T17:46:37.065ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
24172024-07-19T17:46:37.065ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }
24182024-07-19T17:46:37.084ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
24192024-07-19T17:46:37.084ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }
24202024-07-19T17:46:37.104ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
24212024-07-19T17:46:37.104ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }
24222024-07-19T17:46:37.123ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
24232024-07-19T17:46:37.123ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }
24242024-07-19T17:46:37.141ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
24252024-07-19T17:46:37.141ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }
24262024-07-19T17:46:37.160ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
24272024-07-19T17:46:37.160ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }
24282024-07-19T17:46:37.178ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
24292024-07-19T17:46:37.178ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }
24302024-07-19T17:46:37.197ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
24312024-07-19T17:46:37.197ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }
24322024-07-19T17:46:37.216ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
24332024-07-19T17:46:37.216ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/omicron-zones.json file = common/src/ledger.rs:177
24342024-07-19T17:46:37.216ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/omicron-zones.json file = common/src/ledger.rs:177
24352024-07-19T17:46:37.216ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
24362024-07-19T17:46:37.309ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
24372024-07-19T17:46:37.309ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
24382024-07-19T17:46:37.309ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
24392024-07-19T17:46:37.309ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
24402024-07-19T17:46:37.309ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
24412024-07-19T17:46:37.309ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
24422024-07-19T17:46:37.309ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
24432024-07-19T17:46:37.309ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
24442024-07-19T17:46:37.309ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
24452024-07-19T17:46:37.309ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
24462024-07-19T17:46:37.309ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
24472024-07-19T17:46:37.309ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
24482024-07-19T17:46:37.309ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
24492024-07-19T17:46:37.310ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
24502024-07-19T17:46:37.310ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
24512024-07-19T17:46:37.310ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
24522024-07-19T17:46:37.310ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
24532024-07-19T17:46:37.310ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
24542024-07-19T17:46:37.310ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2984 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
24552024-07-19T17:46:37.310ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660", TimeNotSynchronized), ("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b", TimeNotSynchronized), ("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203", TimeNotSynchronized), ("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9", TimeNotSynchronized), ("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603", TimeNotSynchronized), ("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284", TimeNotSynchronized), ("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd", TimeNotSynchronized), ("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b", TimeNotSynchronized), ("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17", TimeNotSynchronized), ("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52", TimeNotSynchronized), ("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1", TimeNotSynchronized), ("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f", TimeNotSynchronized), ("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec", TimeNotSynchronized), ("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7", TimeNotSynchronized), ("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1", TimeNotSynchronized), ("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb", TimeNotSynchronized), ("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957", TimeNotSynchronized), ("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1", TimeNotSynchronized), ("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 478442 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:37752 req_id = b94f954e-136f-45de-adad-3b3570e998a7 response_code = 500 uri = /omicron-zones
24562024-07-19T17:46:37.310ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "b94f954e-136f-45de-adad-3b3570e998a7", "content-length": "124", "date": "Fri, 19 Jul 2024 17:46:36 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "b94f954e-136f-45de-adad-3b3570e998a7" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 11.210331039s sled_address = [fd00:1122:3344:101::1]:12345
24572024-07-19T17:46:43.965ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:57738
24582024-07-19T17:46:43.965ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 30 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:57738 req_id = e353c946-5a80-4750-bf3b-451bfa707ef8 response_code = 200 uri = /baseboard
24592024-07-19T17:46:48.491ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 8ae14f97-bd01-42ad-87d5-23eae9194521, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: bc60810c-eece-4cbd-a0f8-d04271924b7c, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bf3fc707-b452-4d30-a92b-e56b4d4122ee, kind: Service { id: bc60810c-eece-4cbd-a0f8-d04271924b7c }, name: Name("ntp-bc60810c-eece-4cbd-a0f8-d04271924b7c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 196])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: c168b27b-f467-4350-a348-4006ce85a213, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 15be01e5-1765-45f0-b521-da99b395bdb3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: d8a1310d-09c1-4844-91b7-509f93b30cfd, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 59731bd7-ffc9-493f-a33c-a1cf5ba2df30, kind: Service { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1 }, name: Name("external-dns-1eb90f9c-ba99-428e-885b-5fa2df7ae2f1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 32])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 09fdc6fd-8297-436e-903d-b18710599203, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 005b2740-84d5-473e-ae59-aa79fd2fe5b6, kind: Service { id: 09fdc6fd-8297-436e-903d-b18710599203 }, name: Name("external-dns-09fdc6fd-8297-436e-903d-b18710599203"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 6])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: c6edf68c-f558-4f51-9aed-6eb91fb3ad51, kind: Service { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52 }, name: Name("nexus-87d65a87-e1e5-43e4-9cba-b4afd5e9ac52"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 117])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6088410e-a8fc-4d33-b593-6385e863291f, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: e6898038-bf11-41c8-b971-6ff3eed6d38b, kind: Service { id: 6088410e-a8fc-4d33-b593-6385e863291f }, name: Name("nexus-6088410e-a8fc-4d33-b593-6385e863291f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 27])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 742e0330-77de-4723-a145-74751d41c73b, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: ffe6d41d-1349-42b2-a887-49d734eef422, kind: Service { id: 742e0330-77de-4723-a145-74751d41c73b }, name: Name("nexus-742e0330-77de-4723-a145-74751d41c73b"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 199, 184])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8d644923-5967-4351-a1ad-fe8a938f33e1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 740bd3e0-b30c-4033-968a-5451c8d5adcd, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 3286a4ca-fffd-4ed0-9c21-d528e1d23603, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 0b1925c8-3ae5-4372-a3de-213551574fe1, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External } } } }, OmicronZoneConfig { id: 663bd1f7-7a29-484c-bfce-d312646b1660, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External } } } }, OmicronZoneConfig { id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External } } } }, OmicronZoneConfig { id: c6d325c3-cddf-4528-8ba6-a083f6511284, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External } } } }, OmicronZoneConfig { id: c30b7bc1-fd10-4942-8055-ada48d2e3957, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External } } } }, OmicronZoneConfig { id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
24602024-07-19T17:46:48.492ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
24612024-07-19T17:46:48.492ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8ae14f97-bd01-42ad-87d5-23eae9194521, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: InternalDns }, responder: ... }
24622024-07-19T17:46:48.515ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
24632024-07-19T17:46:48.515ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 98b939a3-2514-4766-b854-3b3ddf8b23f9, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: InternalDns }, responder: ... }
24642024-07-19T17:46:48.535ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
24652024-07-19T17:46:48.535ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 8f9c564d-5823-4b85-a0ff-2de7f54b602e, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: InternalDns }, responder: ... }
24662024-07-19T17:46:48.555ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
24672024-07-19T17:46:48.556ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 27c74baf-8069-4296-ac26-2f4818d2ab3a, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24682024-07-19T17:46:48.574ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
24692024-07-19T17:46:48.574ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c168b27b-f467-4350-a348-4006ce85a213, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24702024-07-19T17:46:48.592ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
24712024-07-19T17:46:48.592ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 15be01e5-1765-45f0-b521-da99b395bdb3, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24722024-07-19T17:46:48.609ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
24732024-07-19T17:46:48.610ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: ecfb00b6-cf92-4e9f-ab59-5520a05aa0d0, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24742024-07-19T17:46:48.628ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:404
24752024-07-19T17:46:48.628ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: d8a1310d-09c1-4844-91b7-509f93b30cfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24762024-07-19T17:46:48.645ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
24772024-07-19T17:46:48.645ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
24782024-07-19T17:46:48.664ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:404
24792024-07-19T17:46:48.664ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 09fdc6fd-8297-436e-903d-b18710599203, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: ExternalDns }, responder: ... }
24802024-07-19T17:46:48.682ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:404
24812024-07-19T17:46:48.682ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 2212adda-2fcf-4d59-888c-2223a4fe3f6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
24822024-07-19T17:46:48.700ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
24832024-07-19T17:46:48.700ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 0b1925c8-3ae5-4372-a3de-213551574fe1, dataset_name: DatasetName { pool_name: ZpoolName { id: 988e775a-ddc4-455d-bc4e-f2e3c228cc74 (zpool), kind: External }, kind: Crucible }, responder: ... }
24842024-07-19T17:46:48.718ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
24852024-07-19T17:46:48.718ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 663bd1f7-7a29-484c-bfce-d312646b1660, dataset_name: DatasetName { pool_name: ZpoolName { id: f3be927c-0009-45ce-8f85-0c94f9b67b8f (zpool), kind: External }, kind: Crucible }, responder: ... }
24862024-07-19T17:46:48.737ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
24872024-07-19T17:46:48.737ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 605558ae-0f1e-47a7-b88c-f397548d4ea1, dataset_name: DatasetName { pool_name: ZpoolName { id: ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765 (zpool), kind: External }, kind: Crucible }, responder: ... }
24882024-07-19T17:46:48.756ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
24892024-07-19T17:46:48.756ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: bd7f3d92-4497-48dd-9f51-160ce4d7f4d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 4c50992d-c53d-42b7-a46f-5340e3aee556 (zpool), kind: External }, kind: Crucible }, responder: ... }
24902024-07-19T17:46:48.776ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
24912024-07-19T17:46:48.776ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c6d325c3-cddf-4528-8ba6-a083f6511284, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c02949e-12b1-4d82-9ac3-2efd807aef95 (zpool), kind: External }, kind: Crucible }, responder: ... }
24922024-07-19T17:46:48.794ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
24932024-07-19T17:46:48.796ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 6fa8dbd2-a741-440b-98a2-b60bf983cdc7, dataset_name: DatasetName { pool_name: ZpoolName { id: 765bc25e-72d0-4d30-8cfe-a58b7db5c2e8 (zpool), kind: External }, kind: Crucible }, responder: ... }
24942024-07-19T17:46:48.812ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
24952024-07-19T17:46:48.812ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 696d009b-6d0c-4082-b81b-231c6c85d568 (zpool), kind: External }, kind: Crucible }, responder: ... }
24962024-07-19T17:46:48.830ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
24972024-07-19T17:46:48.830ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: c30b7bc1-fd10-4942-8055-ada48d2e3957, dataset_name: DatasetName { pool_name: ZpoolName { id: 311acf6c-db4a-40cb-bd1f-e33d2bc8747f (zpool), kind: External }, kind: Crucible }, responder: ... }
24982024-07-19T17:46:48.847ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:404
24992024-07-19T17:46:48.847ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:771 request = NewFilesystemRequest { dataset_id: 139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17, dataset_name: DatasetName { pool_name: ZpoolName { id: 41faf794-cdf9-49a0-b529-4c18e335c28c (zpool), kind: External }, kind: Crucible }, responder: ... }
25002024-07-19T17:46:48.865ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
25012024-07-19T17:46:48.865ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/omicron-zones.json file = common/src/ledger.rs:177
25022024-07-19T17:46:48.865ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/omicron-zones.json file = common/src/ledger.rs:177
25032024-07-19T17:46:48.866ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
25042024-07-19T17:46:48.993ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:263
25052024-07-19T17:46:48.994ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
25062024-07-19T17:46:48.994ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
25072024-07-19T17:46:48.994ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
25082024-07-19T17:46:48.995ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:263
25092024-07-19T17:46:49.041ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
25102024-07-19T17:46:49.043ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
25112024-07-19T17:46:49.043ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
25122024-07-19T17:46:49.043ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
25132024-07-19T17:46:49.043ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
25142024-07-19T17:46:49.191ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:263
25152024-07-19T17:46:49.193ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:263
25162024-07-19T17:46:49.243ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:263
25172024-07-19T17:46:49.243ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
25182024-07-19T17:46:49.243ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
25192024-07-19T17:46:49.243ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
25202024-07-19T17:46:49.243ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
25212024-07-19T17:46:49.243ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
25222024-07-19T17:46:49.243ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
25232024-07-19T17:46:49.243ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9 file = illumos-utils/src/zone.rs:316
25242024-07-19T17:46:49.248ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284 file = illumos-utils/src/zone.rs:316
25252024-07-19T17:46:49.252ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957 file = illumos-utils/src/zone.rs:316
25262024-07-19T17:46:49.257ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603 file = illumos-utils/src/zone.rs:316
25272024-07-19T17:46:49.262ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7 file = illumos-utils/src/zone.rs:316
25282024-07-19T17:46:49.267ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b file = illumos-utils/src/zone.rs:316
25292024-07-19T17:46:49.272ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb file = illumos-utils/src/zone.rs:316
25302024-07-19T17:46:49.276ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17 file = illumos-utils/src/zone.rs:316
25312024-07-19T17:46:49.393ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9 file = illumos-utils/src/zone.rs:354
25322024-07-19T17:46:49.397ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284 file = illumos-utils/src/zone.rs:354
25332024-07-19T17:46:49.401ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957 file = illumos-utils/src/zone.rs:354
25342024-07-19T17:46:49.406ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603 file = illumos-utils/src/zone.rs:354
25352024-07-19T17:46:49.410ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7 file = illumos-utils/src/zone.rs:354
25362024-07-19T17:46:49.415ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b file = illumos-utils/src/zone.rs:354
25372024-07-19T17:46:49.420ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb file = illumos-utils/src/zone.rs:354
25382024-07-19T17:46:49.425ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17 file = illumos-utils/src/zone.rs:354
25392024-07-19T17:46:49.432ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1 file = illumos-utils/src/zone.rs:316
25402024-07-19T17:46:49.438ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec file = illumos-utils/src/zone.rs:316
25412024-07-19T17:46:49.443ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd file = illumos-utils/src/zone.rs:316
25422024-07-19T17:46:49.449ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1 file = illumos-utils/src/zone.rs:316
25432024-07-19T17:46:49.455ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660 file = illumos-utils/src/zone.rs:316
25442024-07-19T17:46:49.460ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1 file = illumos-utils/src/zone.rs:316
25452024-07-19T17:46:49.467ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1 file = illumos-utils/src/zone.rs:354
25462024-07-19T17:46:49.473ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec file = illumos-utils/src/zone.rs:354
25472024-07-19T17:46:49.480ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd file = illumos-utils/src/zone.rs:354
25482024-07-19T17:46:49.485ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1 file = illumos-utils/src/zone.rs:354
25492024-07-19T17:46:49.491ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660 file = illumos-utils/src/zone.rs:354
25502024-07-19T17:46:49.498ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1 file = illumos-utils/src/zone.rs:354
25512024-07-19T17:46:49.563ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:280
25522024-07-19T17:46:49.646ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:280
25532024-07-19T17:46:49.772ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:280
25542024-07-19T17:46:49.832ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:280
25552024-07-19T17:46:49.888ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:280
25562024-07-19T17:46:49.890ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:305
25572024-07-19T17:46:49.890ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:197
25582024-07-19T17:46:49.940ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:426 port = Port { inner: PortInner(PortData { name: "opte1", ip: 172.30.2.7, mac: MacAddr6([168, 64, 37, 255, 199, 184]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
25592024-07-19T17:46:49.940ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1210 zone_type = nexus
25602024-07-19T17:46:49.941ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:305
25612024-07-19T17:46:49.941ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:197
25622024-07-19T17:46:49.995ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:426 port = Port { inner: PortInner(PortData { name: "opte2", ip: 172.30.2.5, mac: MacAddr6([168, 64, 37, 255, 163, 117]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
25632024-07-19T17:46:49.995ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1210 zone_type = nexus
25642024-07-19T17:46:49.996ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:305
25652024-07-19T17:46:49.996ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:197
25662024-07-19T17:46:50.067ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:426 port = Port { inner: PortInner(PortData { name: "opte3", ip: 172.30.2.6, mac: MacAddr6([168, 64, 37, 255, 171, 27]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
25672024-07-19T17:46:50.067ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1210 zone_type = nexus
25682024-07-19T17:46:50.068ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:305
25692024-07-19T17:46:50.068ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:197
25702024-07-19T17:46:50.120ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:426 port = Port { inner: PortInner(PortData { name: "opte4", ip: 172.30.1.6, mac: MacAddr6([168, 64, 37, 255, 187, 6]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
25712024-07-19T17:46:50.120ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1210 zone_type = external_dns
25722024-07-19T17:46:50.122ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:305
25732024-07-19T17:46:50.122ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:197
25742024-07-19T17:46:50.176ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:426 port = Port { inner: PortInner(PortData { name: "opte5", ip: 172.30.1.5, mac: MacAddr6([168, 64, 37, 255, 175, 32]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
25752024-07-19T17:46:50.176ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1210 zone_type = external_dns
25762024-07-19T17:46:50.177ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:83
25772024-07-19T17:46:50.177ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:90
25782024-07-19T17:46:50.179ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:83
25792024-07-19T17:46:50.179ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:90
25802024-07-19T17:46:50.179ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:83
25812024-07-19T17:46:50.179ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:90
25822024-07-19T17:46:50.179ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:83
25832024-07-19T17:46:50.179ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:90
25842024-07-19T17:46:50.180ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:83
25852024-07-19T17:46:50.180ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:90
25862024-07-19T17:46:50.182ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:118
25872024-07-19T17:46:50.183ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
25882024-07-19T17:46:50.183ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:118
25892024-07-19T17:46:50.231ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
25902024-07-19T17:46:50.231ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:118
25912024-07-19T17:46:50.236ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:118
25922024-07-19T17:46:50.239ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
25932024-07-19T17:46:50.239ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
25942024-07-19T17:46:50.239ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:118
25952024-07-19T17:46:50.287ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203 file = illumos-utils/src/zone.rs:316
25962024-07-19T17:46:50.328ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
25972024-07-19T17:46:50.441ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f file = illumos-utils/src/zone.rs:316
25982024-07-19T17:46:50.448ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203 file = illumos-utils/src/zone.rs:354
25992024-07-19T17:46:50.453ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1 file = illumos-utils/src/zone.rs:316
26002024-07-19T17:46:50.460ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_742e0330-77de-4723-a145-74751d41c73b file = illumos-utils/src/zone.rs:316
26012024-07-19T17:46:50.467ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52 file = illumos-utils/src/zone.rs:316
26022024-07-19T17:46:50.473ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f file = illumos-utils/src/zone.rs:354
26032024-07-19T17:46:50.483ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1 file = illumos-utils/src/zone.rs:354
26042024-07-19T17:46:50.494ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52 file = illumos-utils/src/zone.rs:354
26052024-07-19T17:46:50.500ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_742e0330-77de-4723-a145-74751d41c73b file = illumos-utils/src/zone.rs:354
26062024-07-19T17:46:53.206ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService13"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::10"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::10"/> <propval type="astring" name="listen_port" value="17000"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
26072024-07-19T17:46:53.207ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:465 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
26082024-07-19T17:46:53.301ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService12"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::19"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_311acf6c-db4a-40cb-bd1f-e33d2bc8747f/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::19"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="b764a43e-0fb7-4d29-af0f-fc18b1577aa8"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
26092024-07-19T17:46:53.302ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:465 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
26102024-07-19T17:46:53.357ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService16"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::18"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_696d009b-6d0c-4082-b81b-231c6c85d568/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::18"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="a72c4ab6-87b4-444e-986c-50cd16017338"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
26112024-07-19T17:46:53.358ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:465 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
26122024-07-19T17:46:53.369ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService11"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::16"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_7c02949e-12b1-4d82-9ac3-2efd807aef95/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::16"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="6a028fb0-62cf-453e-bc98-c7aba8ca7a96"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
26132024-07-19T17:46:53.369ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:465 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
26142024-07-19T17:46:53.377ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService10"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::15"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_4c50992d-c53d-42b7-a46f-5340e3aee556/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::15"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="ef250c8f-b0af-4c48-8d04-49bf2de5a7c1"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
26152024-07-19T17:46:53.377ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:465 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
26162024-07-19T17:46:53.418ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService22"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::13"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_f3be927c-0009-45ce-8f85-0c94f9b67b8f/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::13"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="558b5f39-dc42-4eac-9a42-75d0203d5b65"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
26172024-07-19T17:46:53.418ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:465 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
26182024-07-19T17:46:53.461ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService17"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::1a"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_41faf794-cdf9-49a0-b529-4c18e335c28c/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::1a"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="2c02dce3-bc32-40b2-a2fb-6b36dfac025c"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
26192024-07-19T17:46:53.462ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:465 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
26202024-07-19T17:46:53.470ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService14"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::17"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_765bc25e-72d0-4d30-8cfe-a58b7db5c2e8/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::17"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="61560500-7532-4dce-b016-9eefb72e65f3"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
26212024-07-19T17:46:53.470ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:465 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
26222024-07-19T17:46:53.509ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603") failed: Property not found. retry in 42.671196ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
26232024-07-19T17:46:53.513ZINFOSledAgent (ServiceManager): Profile for oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService18"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::d"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/oximeter"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="id" value="8d644923-5967-4351-a1ad-fe8a938f33e1"/> <propval type="astring" name="address" value="[fd00:1122:3344:101::d]:12223"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
26242024-07-19T17:46:53.514ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:465 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
26252024-07-19T17:46:53.524ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService21"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::12"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_988e775a-ddc4-455d-bc4e-f2e3c228cc74/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::12"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="870f5363-ba0f-4d2c-86da-804485930453"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
26262024-07-19T17:46:53.524ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:465 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
26272024-07-19T17:46:53.544ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957") failed: Property not found. retry in 49.586827ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
26282024-07-19T17:46:53.565ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603") failed: Property not found. retry in 133.276912ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
26292024-07-19T17:46:53.583ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284") failed: Property not found. retry in 27.674166ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
26302024-07-19T17:46:53.595ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb") failed: Property not found. retry in 65.300412ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
26312024-07-19T17:46:53.606ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9") failed: Property not found. retry in 32.90706ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
26322024-07-19T17:46:53.617ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957") failed: Property not found. retry in 128.500648ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
26332024-07-19T17:46:53.631ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284") failed: Property not found. retry in 101.459504ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
26342024-07-19T17:46:53.745ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9") failed: Property not found. retry in 116.832572ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
26352024-07-19T17:46:53.887ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284") failed: Property not found. retry in 253.062913ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
26362024-07-19T17:46:53.954ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957") failed: Property not found. retry in 265.605423ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
26372024-07-19T17:46:56.461ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603") failed: Property not found. retry in 260.559496ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
26382024-07-19T17:46:56.530ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb") failed: Property not found. retry in 60.492294ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
26392024-07-19T17:46:56.659ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17") failed: Property not found. retry in 26.463018ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
26402024-07-19T17:46:56.924ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660") failed: Property not found. retry in 34.817939ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
26412024-07-19T17:46:57.023ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9") failed: Property not found. retry in 218.893169ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
26422024-07-19T17:46:57.127ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284") failed: Property not found. retry in 333.13421ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
26432024-07-19T17:46:57.152ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957") failed: Property not found. retry in 377.990133ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
26442024-07-19T17:46:57.174ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603") failed: Property not found. retry in 288.696625ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
26452024-07-19T17:46:57.201ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7") failed: Property not found. retry in 30.461889ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
26462024-07-19T17:46:57.276ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb") failed: Property not found. retry in 228.830816ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
26472024-07-19T17:46:57.448ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17") failed: Property not found. retry in 85.674858ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
26482024-07-19T17:46:57.526ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1") failed: Property not found. retry in 63.310489ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
26492024-07-19T17:46:57.600ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1") failed: Property not found. retry in 73.418329ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
26502024-07-19T17:46:57.620ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660") failed: Property not found. retry in 56.554687ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
26512024-07-19T17:46:57.644ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9") failed: Property not found. retry in 545.23398ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
26522024-07-19T17:46:57.659ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284") failed: Property not found. retry in 751.606524ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
26532024-07-19T17:46:57.674ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957") failed: Property not found. retry in 678.077683ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
26542024-07-19T17:46:57.689ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603") failed: Property not found. retry in 572.378501ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
26552024-07-19T17:46:57.707ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7") failed: Property not found. retry in 61.334974ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
26562024-07-19T17:46:57.723ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb") failed: Property not found. retry in 439.185821ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
26572024-07-19T17:46:57.738ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17") failed: Property not found. retry in 186.006606ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
26582024-07-19T17:46:57.754ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1") failed: Property not found. retry in 91.210483ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
26592024-07-19T17:46:57.756ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService19"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::11"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::11"/> <propval type="astring" name="listen_port" value="17000"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
26602024-07-19T17:46:57.757ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService20"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::f"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::f"/> <propval type="astring" name="listen_port" value="17000"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
26612024-07-19T17:46:57.774ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1") failed: Property not found. retry in 131.38106ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
26622024-07-19T17:46:57.789ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660") failed: Property not found. retry in 181.100652ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
26632024-07-19T17:46:57.803ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7") failed: Property not found. retry in 213.117473ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
26642024-07-19T17:46:57.803ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:465 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
26652024-07-19T17:46:57.812ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:465 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
26662024-07-19T17:46:57.829ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService23"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::14"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::14"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="af78d7bf-8b90-4538-b939-ba9b7d0e749c"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
26672024-07-19T17:46:57.829ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:465 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
26682024-07-19T17:46:57.862ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1") failed: Property not found. retry in 207.912431ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
26692024-07-19T17:46:57.925ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1") failed: Property not found. retry in 284.904366ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
26702024-07-19T17:46:57.942ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17") failed: Property not found. retry in 217.528863ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
26712024-07-19T17:46:57.992ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660") failed: Property not found. retry in 437.962178ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
26722024-07-19T17:46:58.033ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7") failed: Property not found. retry in 283.244365ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
26732024-07-19T17:46:58.086ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1") failed: Property not found. retry in 228.191289ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
26742024-07-19T17:46:58.176ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17") failed: Property not found. retry in 1.197053231s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
26752024-07-19T17:46:58.192ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb") failed: Property not found. retry in 1.007683317s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
26762024-07-19T17:46:58.206ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9") failed: Property not found. retry in 655.531247ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
26772024-07-19T17:46:58.220ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec") failed: Property not found. retry in 58.172534ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
26782024-07-19T17:46:58.232ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1") failed: Property not found. retry in 64.021528ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
26792024-07-19T17:46:58.244ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd") failed: Property not found. retry in 72.925618ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
26802024-07-19T17:46:58.256ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1") failed: Property not found. retry in 285.579094ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
26812024-07-19T17:46:58.274ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603") failed: Property not found. retry in 798.411374ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
26822024-07-19T17:46:58.289ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec") failed: Property not found. retry in 118.799427ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
26832024-07-19T17:46:58.321ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1") failed: Property not found. retry in 101.775719ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
26842024-07-19T17:46:58.345ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7") failed: Property not found. retry in 1.049237261s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
26852024-07-19T17:46:58.374ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1") failed: Property not found. retry in 711.603215ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
26862024-07-19T17:46:58.393ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd") failed: Property not found. retry in 73.150747ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
26872024-07-19T17:46:58.414ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957") failed: Property not found. retry in 1.142180569s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
26882024-07-19T17:46:58.432ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284") failed: Property not found. retry in 1.101909839s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
26892024-07-19T17:46:58.448ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec") failed: Property not found. retry in 287.900089ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
26902024-07-19T17:46:58.467ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660") failed: Property not found. retry in 1.092994235s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
26912024-07-19T17:46:58.481ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1") failed: Property not found. retry in 225.024608ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
26922024-07-19T17:46:58.495ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd") failed: Property not found. retry in 279.304853ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
26932024-07-19T17:46:58.507ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 19.258354202s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
26942024-07-19T17:46:58.564ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1") failed: Property not found. retry in 494.117672ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
26952024-07-19T17:46:58.790ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1") failed: Property not found. retry in 406.788016ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
26962024-07-19T17:46:58.812ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec") failed: Property not found. retry in 330.416092ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
26972024-07-19T17:46:58.833ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd") failed: Property not found. retry in 599.106685ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
26982024-07-19T17:46:58.880ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9") failed: Property not found. retry in 500.438886ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
26992024-07-19T17:46:59.079ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1") failed: Property not found. retry in 516.893883ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
27002024-07-19T17:46:59.102ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603") failed: Property not found. retry in 557.230679ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
27012024-07-19T17:46:59.119ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1") failed: Property not found. retry in 1.228958481s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
27022024-07-19T17:46:59.158ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec") failed: Property not found. retry in 794.317636ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
27032024-07-19T17:46:59.212ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1") failed: Property not found. retry in 581.700211ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
27042024-07-19T17:46:59.227ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb") failed: Property not found. retry in 567.980403ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
27052024-07-19T17:46:59.389ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17") failed: Property not found. retry in 1.416209326s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
27062024-07-19T17:46:59.404ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9") failed: Property not found. retry in 860.330191ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
27072024-07-19T17:46:59.420ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7") failed: Property not found. retry in 1.090737903s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
27082024-07-19T17:46:59.449ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd") failed: Property not found. retry in 556.403996ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
27092024-07-19T17:46:59.547ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284") failed: Property not found. retry in 838.865868ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
27102024-07-19T17:46:59.568ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957") failed: Property not found. retry in 813.349569ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
27112024-07-19T17:46:59.580ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660") failed: Property not found. retry in 1.345370503s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
27122024-07-19T17:46:59.581ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService26"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::8"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="opte5"/> <propval type="astring" name="gateway" value="172.30.1.1"/> <propval type="astring" name="ip" value="172.30.1.5"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::8]:5353"/> <propval type="astring" name="dns_address" value="172.30.1.5:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
27132024-07-19T17:46:59.581ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService24"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::9"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="opte4"/> <propval type="astring" name="gateway" value="172.30.1.1"/> <propval type="astring" name="ip" value="172.30.1.6"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::9]:5353"/> <propval type="astring" name="dns_address" value="172.30.1.6:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
27142024-07-19T17:46:59.581ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:465 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
27152024-07-19T17:46:59.588ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:465 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
27162024-07-19T17:46:59.608ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1") failed: Property not found. retry in 1.377391407s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
27172024-07-19T17:46:59.672ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603") failed: Property not found. retry in 1.447664736s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
27182024-07-19T17:46:59.813ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1") failed: Property not found. retry in 1.295998986s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
27192024-07-19T17:46:59.828ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1") failed: Property not found. retry in 56.863795ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
27202024-07-19T17:46:59.846ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb") failed: Property not found. retry in 543.157729ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
27212024-07-19T17:46:59.860ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203") failed: Property not found. retry in 73.410066ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
27222024-07-19T17:46:59.902ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1") failed: Property not found. retry in 69.807334ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
27232024-07-19T17:46:59.986ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203") failed: Property not found. retry in 121.782092ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
27242024-07-19T17:47:00.013ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1") failed: Property not found. retry in 168.837796ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
27252024-07-19T17:47:00.046ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec") failed: Property not found. retry in 1.395804975s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
27262024-07-19T17:47:00.070ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd") failed: Property not found. retry in 975.233418ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
27272024-07-19T17:47:00.135ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203") failed: Property not found. retry in 112.119358ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
27282024-07-19T17:47:00.202ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1") failed: Property not found. retry in 572.42778ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
27292024-07-19T17:47:00.372ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203") failed: Property not found. retry in 204.606199ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
27302024-07-19T17:47:00.400ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9") failed: Property not found. retry in 758.353744ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
27312024-07-19T17:47:00.461ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1") failed: Property not found. retry in 1.370846755s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
27322024-07-19T17:47:00.547ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284") failed: Property not found. retry in 769.332724ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
27332024-07-19T17:47:00.570ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957") failed: Property not found. retry in 746.931419ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
27342024-07-19T17:47:00.588ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7") failed: Property not found. retry in 1.120380065s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
27352024-07-19T17:47:00.605ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb") failed: Property not found. retry in 582.269227ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
27362024-07-19T17:47:00.620ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203") failed: Property not found. retry in 937.646348ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
27372024-07-19T17:47:00.790ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1") failed: Property not found. retry in 744.638989ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
27382024-07-19T17:47:00.819ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17") failed: Property not found. retry in 1.014483368s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
27392024-07-19T17:47:00.941ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660") failed: Property not found. retry in 975.57683ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
27402024-07-19T17:47:01.003ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1") failed: Property not found. retry in 644.881737ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
27412024-07-19T17:47:01.059ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd") failed: Property not found. retry in 1.400078817s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
27422024-07-19T17:47:01.121ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1") failed: Property not found. retry in 1.435191729s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
27432024-07-19T17:47:01.133ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603") failed: Property not found. retry in 858.593535ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
27442024-07-19T17:47:01.169ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9") failed: Property not found. retry in 1.49370871s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
27452024-07-19T17:47:01.199ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb") failed: Property not found. retry in 1.377637436s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
27462024-07-19T17:47:01.281ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_742e0330-77de-4723-a145-74751d41c73b: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService27"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::c"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="opte1"/> <propval type="astring" name="gateway" value="172.30.2.1"/> <propval type="astring" name="ip" value="172.30.2.7"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
27472024-07-19T17:47:01.281ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService28"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::a"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="opte2"/> <propval type="astring" name="gateway" value="172.30.2.1"/> <propval type="astring" name="ip" value="172.30.2.5"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
27482024-07-19T17:47:01.282ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService25"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::b"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="opte3"/> <propval type="astring" name="gateway" value="172.30.2.1"/> <propval type="astring" name="ip" value="172.30.2.6"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
27492024-07-19T17:47:01.282ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:465 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
27502024-07-19T17:47:01.289ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:465 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
27512024-07-19T17:47:01.300ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:465 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
27522024-07-19T17:47:01.328ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284") failed: Property not found. retry in 615.071809ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
27532024-07-19T17:47:01.339ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957") failed: Property not found. retry in 573.491992ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
27542024-07-19T17:47:01.456ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec") failed: Property not found. retry in 1.441529806s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
27552024-07-19T17:47:01.611ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1") failed: Property not found. retry in 1.163177448s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
27562024-07-19T17:47:01.639ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b") failed: Property not found. retry in 61.963012ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
27572024-07-19T17:47:01.663ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52") failed: Property not found. retry in 60.681008ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
27582024-07-19T17:47:01.684ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203") failed: Property not found. retry in 1.013960173s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
27592024-07-19T17:47:01.702ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f") failed: Property not found. retry in 30.005659ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
27602024-07-19T17:47:01.721ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1") failed: Property not found. retry in 644.520072ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
27612024-07-19T17:47:01.749ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b") failed: Property not found. retry in 57.379735ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
27622024-07-19T17:47:01.774ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7") failed: Property not found. retry in 1.437060697s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
27632024-07-19T17:47:01.809ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52") failed: Property not found. retry in 79.781931ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
27642024-07-19T17:47:01.829ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f") failed: Property not found. retry in 106.355821ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
27652024-07-19T17:47:01.857ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b") failed: Property not found. retry in 143.899668ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
27662024-07-19T17:47:01.883ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17") failed: Property not found. retry in 1.105750712s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
27672024-07-19T17:47:01.904ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1") failed: Property not found. retry in 725.450162ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
27682024-07-19T17:47:01.929ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52") failed: Property not found. retry in 210.804334ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
27692024-07-19T17:47:01.953ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660") failed: Property not found. retry in 837.766378ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
27702024-07-19T17:47:01.970ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284") failed: Property not found. retry in 1.036437053s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
27712024-07-19T17:47:01.986ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957") failed: Property not found. retry in 987.64218ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
27722024-07-19T17:47:02.005ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f") failed: Property not found. retry in 206.639489ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
27732024-07-19T17:47:02.040ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b") failed: Property not found. retry in 226.096906ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
27742024-07-19T17:47:02.064ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603") failed: Property not found. retry in 810.729514ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
27752024-07-19T17:47:02.069ZINFOSledAgent (ServiceManager): Profile for oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService15"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::e"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::e"/> <propval type="astring" name="listen_port" value="8123"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
27762024-07-19T17:47:02.069ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:465 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
27772024-07-19T17:47:02.189ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52") failed: Property not found. retry in 505.805452ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
27782024-07-19T17:47:02.227ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f") failed: Property not found. retry in 282.444076ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
27792024-07-19T17:47:02.266ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b") failed: Property not found. retry in 1.141627052s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
27802024-07-19T17:47:02.359ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b") failed: Property not found. retry in 34.168512ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
27812024-07-19T17:47:02.379ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1") failed: Property not found. retry in 771.385474ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
27822024-07-19T17:47:02.403ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b") failed: Property not found. retry in 62.294285ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
27832024-07-19T17:47:02.472ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd") failed: Property not found. retry in 1.147178293s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
27842024-07-19T17:47:02.482ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b") failed: Property not found. retry in 260.332708ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
27852024-07-19T17:47:02.519ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f") failed: Property not found. retry in 1.09342023s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
27862024-07-19T17:47:02.567ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1") failed: Property not found. retry in 519.40274ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
27872024-07-19T17:47:02.589ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb") failed: Property not found. retry in 665.710402ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
27882024-07-19T17:47:02.661ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1") failed: Property not found. retry in 626.02901ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
27892024-07-19T17:47:02.697ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9") failed: Property not found. retry in 1.330352943s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
27902024-07-19T17:47:02.710ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52") failed: Property not found. retry in 962.838838ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
27912024-07-19T17:47:02.728ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203") failed: Property not found. retry in 587.008173ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
27922024-07-19T17:47:02.761ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b") failed: Property not found. retry in 486.380233ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
27932024-07-19T17:47:02.791ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1") failed: Property not found. retry in 514.479278ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
27942024-07-19T17:47:02.804ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660") failed: Property not found. retry in 625.34014ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
27952024-07-19T17:47:02.890ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603") failed: Property not found. retry in 871.780212ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
27962024-07-19T17:47:02.910ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec") failed: Property not found. retry in 737.352229ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
27972024-07-19T17:47:02.988ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957") failed: Property not found. retry in 729.669749ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
27982024-07-19T17:47:03.003ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17") failed: Property not found. retry in 670.234051ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
27992024-07-19T17:47:03.021ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284") failed: Property not found. retry in 1.189073053s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
28002024-07-19T17:47:03.100ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1") failed: Property not found. retry in 542.964362ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
28012024-07-19T17:47:03.164ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1") failed: Property not found. retry in 1.381790167s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
28022024-07-19T17:47:03.226ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7") failed: Property not found. retry in 623.900142ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
28032024-07-19T17:47:03.260ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b") failed: Property not found. retry in 1.00240275s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
28042024-07-19T17:47:03.274ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb") failed: Property not found. retry in 548.007133ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
28052024-07-19T17:47:03.299ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1") failed: Property not found. retry in 993.464355ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
28062024-07-19T17:47:03.320ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1") failed: Property not found. retry in 1.04995281s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
28072024-07-19T17:47:03.335ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203") failed: Property not found. retry in 590.803133ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
28082024-07-19T17:47:03.421ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b") failed: Property not found. retry in 932.881571ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
28092024-07-19T17:47:03.441ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660") failed: Property not found. retry in 554.454378ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
28102024-07-19T17:47:03.626ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f") failed: Property not found. retry in 1.177682477s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
28112024-07-19T17:47:03.639ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd") failed: Property not found. retry in 1.30799087s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
28122024-07-19T17:47:03.656ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1") failed: Property not found. retry in 1.490811262s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
28132024-07-19T17:47:03.667ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec") failed: Property not found. retry in 1.027197631s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
28142024-07-19T17:47:03.685ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17") failed: Property not found. retry in 698.995319ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
28152024-07-19T17:47:03.698ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52") failed: Property not found. retry in 1.063966014s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
28162024-07-19T17:47:03.732ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957") failed: Property not found. retry in 831.637317ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
28172024-07-19T17:47:03.775ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603") failed: Property not found. retry in 662.521748ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
28182024-07-19T17:47:03.836ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb") failed: Property not found. retry in 820.005348ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
28192024-07-19T17:47:03.864ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7") failed: Property not found. retry in 1.434552354s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
28202024-07-19T17:47:03.939ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203") failed: Property not found. retry in 745.87054ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
28212024-07-19T17:47:04.012ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660") failed: Property not found. retry in 876.486731ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
28222024-07-19T17:47:04.041ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9") failed: Property not found. retry in 1.162644362s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
28232024-07-19T17:47:04.224ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284") failed: Property not found. retry in 803.296118ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
28242024-07-19T17:47:04.277ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b") failed: Property not found. retry in 1.212643552s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
28252024-07-19T17:47:04.306ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1") failed: Property not found. retry in 920.185484ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
28262024-07-19T17:47:04.369ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b") failed: Property not found. retry in 1.17941535s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
28272024-07-19T17:47:04.386ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1") failed: Property not found. retry in 976.850958ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
28282024-07-19T17:47:04.400ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17") failed: Property not found. retry in 973.957083ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
28292024-07-19T17:47:04.456ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603") failed: Property not found. retry in 959.079993ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
28302024-07-19T17:47:04.560ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1") failed: Property not found. retry in 763.724829ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
28312024-07-19T17:47:04.578ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957") failed: Property not found. retry in 654.639514ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
28322024-07-19T17:47:04.670ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb") failed: Property not found. retry in 1.149114407s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
28332024-07-19T17:47:04.700ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203") failed: Property not found. retry in 1.367798545s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
28342024-07-19T17:47:04.714ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec") failed: Property not found. retry in 1.195053839s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
28352024-07-19T17:47:04.780ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52") failed: Property not found. retry in 1.466575814s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
28362024-07-19T17:47:04.820ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f") failed: Property not found. retry in 585.588943ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
28372024-07-19T17:47:04.902ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660") failed: Property not found. retry in 1.302055956s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
28382024-07-19T17:47:04.961ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd") failed: Property not found. retry in 1.331348029s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
28392024-07-19T17:47:05.059ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284") failed: Property not found. retry in 1.151995214s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
28402024-07-19T17:47:05.163ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1") failed: Property not found. retry in 1.385555209s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
28412024-07-19T17:47:05.219ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9") failed: Property not found. retry in 1.445896123s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
28422024-07-19T17:47:05.242ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1") failed: Property not found. retry in 1.179287276s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
28432024-07-19T17:47:05.258ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957") failed: Property not found. retry in 1.412339047s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
28442024-07-19T17:47:05.311ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7") failed: Property not found. retry in 783.826625ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
28452024-07-19T17:47:05.337ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1") failed: Property not found. retry in 540.452191ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
28462024-07-19T17:47:05.378ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1") failed: Property not found. retry in 568.305554ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
28472024-07-19T17:47:05.390ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17") failed: Property not found. retry in 809.970625ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
28482024-07-19T17:47:05.420ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f") failed: Property not found. retry in 1.092951437s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
28492024-07-19T17:47:05.435ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603") failed: Property not found. retry in 978.591102ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
28502024-07-19T17:47:05.504ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b") failed: Property not found. retry in 837.756052ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
28512024-07-19T17:47:05.563ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b") failed: Property not found. retry in 1.072748893s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
28522024-07-19T17:47:05.834ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb") failed: Property not found. retry in 1.279091631s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
28532024-07-19T17:47:05.890ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1") failed: Property not found. retry in 706.286454ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
28542024-07-19T17:47:05.923ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec") failed: Property not found. retry in 545.952523ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
28552024-07-19T17:47:05.956ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1") failed: Property not found. retry in 813.155803ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
28562024-07-19T17:47:06.080ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203") failed: Property not found. retry in 1.011182494s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
28572024-07-19T17:47:06.108ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7") failed: Property not found. retry in 887.609051ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
28582024-07-19T17:47:06.211ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17") failed: Property not found. retry in 658.028116ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
28592024-07-19T17:47:06.221ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284") failed: Property not found. retry in 798.547606ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
28602024-07-19T17:47:06.231ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660") failed: Property not found. retry in 1.424294212s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
28612024-07-19T17:47:06.259ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52") failed: Property not found. retry in 866.181632ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
28622024-07-19T17:47:06.305ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd") failed: Property not found. retry in 1.134558603s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
28632024-07-19T17:47:06.357ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b") failed: Property not found. retry in 889.470256ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
28642024-07-19T17:47:06.424ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603") failed: Property not found. retry in 1.19338893s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
28652024-07-19T17:47:06.437ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1") failed: Property not found. retry in 1.493925848s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
28662024-07-19T17:47:06.483ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec") failed: Property not found. retry in 706.910091ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
28672024-07-19T17:47:06.524ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f") failed: Property not found. retry in 534.821142ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
28682024-07-19T17:47:06.561ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1") failed: Property not found. retry in 1.203802686s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
28692024-07-19T17:47:06.611ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1") failed: Property not found. retry in 867.016722ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
28702024-07-19T17:47:06.651ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b") failed: Property not found. retry in 781.667191ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
28712024-07-19T17:47:06.677ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9") failed: Property not found. retry in 977.198666ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
28722024-07-19T17:47:06.687ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957") failed: Property not found. retry in 1.312689384s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
28732024-07-19T17:47:06.784ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1") failed: Property not found. retry in 891.817144ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
28742024-07-19T17:47:06.882ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17") failed: Property not found. retry in 753.900992ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
28752024-07-19T17:47:07.007ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7") failed: Property not found. retry in 648.933105ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
28762024-07-19T17:47:07.033ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284") failed: Property not found. retry in 1.29672065s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
28772024-07-19T17:47:07.069ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f") failed: Property not found. retry in 949.879608ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
28782024-07-19T17:47:07.104ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203") failed: Property not found. retry in 614.747368ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
28792024-07-19T17:47:07.124ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb") failed: Property not found. retry in 1.466004798s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
28802024-07-19T17:47:07.138ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52") failed: Property not found. retry in 1.116020717s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
28812024-07-19T17:47:07.204ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec") failed: Property not found. retry in 905.399815ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
28822024-07-19T17:47:07.261ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b") failed: Property not found. retry in 915.983867ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
28832024-07-19T17:47:07.448ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b") failed: Property not found. retry in 1.395155264s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
28842024-07-19T17:47:07.459ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd") failed: Property not found. retry in 1.333998406s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
28852024-07-19T17:47:07.490ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1") failed: Property not found. retry in 1.029020056s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
28862024-07-19T17:47:07.631ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603") failed: Property not found. retry in 1.082828029s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
28872024-07-19T17:47:07.649ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17") failed: Property not found. retry in 995.085262ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
28882024-07-19T17:47:07.666ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9") failed: Property not found. retry in 1.137696743s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
28892024-07-19T17:47:07.678ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7") failed: Property not found. retry in 679.481649ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
28902024-07-19T17:47:07.703ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660") failed: Property not found. retry in 900.733372ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
28912024-07-19T17:47:07.718ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1") failed: Property not found. retry in 702.161582ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
28922024-07-19T17:47:07.730ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203") failed: Property not found. retry in 1.008025511s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
28932024-07-19T17:47:07.778ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1") failed: Property not found. retry in 1.316947314s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
28942024-07-19T17:47:07.944ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1") failed: Property not found. retry in 1.48015068s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
28952024-07-19T17:47:08.012ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957") failed: Property not found. retry in 1.435641241s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
28962024-07-19T17:47:08.031ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f") failed: Property not found. retry in 1.424308585s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
28972024-07-19T17:47:08.123ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec") failed: Property not found. retry in 623.867117ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
28982024-07-19T17:47:08.190ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b") failed: Property not found. retry in 584.104404ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
28992024-07-19T17:47:08.267ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52") failed: Property not found. retry in 560.686859ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
29002024-07-19T17:47:08.342ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284") failed: Property not found. retry in 1.015272536s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
29012024-07-19T17:47:08.371ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7") failed: Property not found. retry in 1.383025486s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
29022024-07-19T17:47:08.434ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1") failed: Property not found. retry in 801.617388ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
29032024-07-19T17:47:08.532ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1") failed: Property not found. retry in 1.464208103s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
29042024-07-19T17:47:08.606ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb") failed: Property not found. retry in 933.665929ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
29052024-07-19T17:47:08.619ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660") failed: Property not found. retry in 589.487353ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
29062024-07-19T17:47:08.656ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17") failed: Property not found. retry in 969.865887ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
29072024-07-19T17:47:08.727ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603") failed: Property not found. retry in 1.010621615s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
29082024-07-19T17:47:08.752ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203") failed: Property not found. retry in 1.008160998s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
29092024-07-19T17:47:08.764ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec") failed: Property not found. retry in 1.113585083s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
29102024-07-19T17:47:08.788ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b") failed: Property not found. retry in 1.08864743s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
29112024-07-19T17:47:08.813ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd") failed: Property not found. retry in 1.495547038s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
29122024-07-19T17:47:08.822ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9") failed: Property not found. retry in 1.400957096s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
29132024-07-19T17:47:08.847ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52") failed: Property not found. retry in 1.075987383s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
29142024-07-19T17:47:08.863ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b") failed: Property not found. retry in 556.537372ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
29152024-07-19T17:47:09.108ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1") failed: Property not found. retry in 569.123167ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
29162024-07-19T17:47:09.221ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660") failed: Property not found. retry in 610.651086ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
29172024-07-19T17:47:09.248ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1") failed: Property not found. retry in 894.271428ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
29182024-07-19T17:47:09.368ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284") failed: Property not found. retry in 912.063346ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
29192024-07-19T17:47:09.430ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b") failed: Property not found. retry in 1.025857981s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
29202024-07-19T17:47:09.439ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1") failed: Property not found. retry in 644.205606ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
29212024-07-19T17:47:09.485ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957") failed: Property not found. retry in 679.108844ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
29222024-07-19T17:47:09.494ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f") failed: Property not found. retry in 692.777912ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
29232024-07-19T17:47:09.549ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb") failed: Property not found. retry in 1.03476986s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
29242024-07-19T17:47:09.635ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17") failed: Property not found. retry in 1.127924977s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
29252024-07-19T17:47:09.687ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1") failed: Property not found. retry in 1.07559805s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
29262024-07-19T17:47:09.747ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603") failed: Property not found. retry in 890.05945ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
29272024-07-19T17:47:09.762ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7") failed: Property not found. retry in 919.193269ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
29282024-07-19T17:47:09.769ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203") failed: Property not found. retry in 1.217117288s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
29292024-07-19T17:47:09.842ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660") failed: Property not found. retry in 1.092358996s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
29302024-07-19T17:47:09.888ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b") failed: Property not found. retry in 967.043175ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
29312024-07-19T17:47:09.899ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec") failed: Property not found. retry in 1.087131737s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
29322024-07-19T17:47:09.934ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52") failed: Property not found. retry in 529.434344ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
29332024-07-19T17:47:10.007ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1") failed: Property not found. retry in 1.108730039s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
29342024-07-19T17:47:10.093ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1") failed: Property not found. retry in 1.23335225s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
29352024-07-19T17:47:10.150ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1") failed: Property not found. retry in 1.100172911s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
29362024-07-19T17:47:10.174ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957") failed: Property not found. retry in 900.608734ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
29372024-07-19T17:47:10.197ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f") failed: Property not found. retry in 1.165934719s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
29382024-07-19T17:47:10.233ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9") failed: Property not found. retry in 1.431754448s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
29392024-07-19T17:47:10.291ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284") failed: Property not found. retry in 976.684379ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
29402024-07-19T17:47:10.318ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd") failed: Property not found. retry in 1.050944033s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
29412024-07-19T17:47:10.467ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b") failed: Property not found. retry in 1.078131109s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
29422024-07-19T17:47:10.478ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52") failed: Property not found. retry in 1.147461742s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
29432024-07-19T17:47:10.594ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb") failed: Property not found. retry in 846.86342ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
29442024-07-19T17:47:10.647ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603") failed: Property not found. retry in 1.224681638s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
29452024-07-19T17:47:10.691ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7") failed: Property not found. retry in 1.028499076s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
29462024-07-19T17:47:10.775ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17") failed: Property not found. retry in 1.229944873s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
29472024-07-19T17:47:10.785ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1") failed: Property not found. retry in 1.330660695s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
29482024-07-19T17:47:10.864ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b") failed: Property not found. retry in 847.929396ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
29492024-07-19T17:47:10.945ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660") failed: Property not found. retry in 1.141466553s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
29502024-07-19T17:47:10.995ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec") failed: Property not found. retry in 696.62597ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
29512024-07-19T17:47:11.003ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203") failed: Property not found. retry in 1.356465877s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
29522024-07-19T17:47:11.083ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957") failed: Property not found. retry in 1.10926155s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
29532024-07-19T17:47:11.123ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1") failed: Property not found. retry in 938.631896ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
29542024-07-19T17:47:11.259ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1") failed: Property not found. retry in 1.06026547s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
29552024-07-19T17:47:11.279ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284") failed: Property not found. retry in 1.47368444s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
29562024-07-19T17:47:11.335ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1") failed: Property not found. retry in 659.001009ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
29572024-07-19T17:47:11.371ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f") failed: Property not found. retry in 1.227875635s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
29582024-07-19T17:47:11.379ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd") failed: Property not found. retry in 827.677ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
29592024-07-19T17:47:11.449ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb") failed: Property not found. retry in 961.680138ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
29602024-07-19T17:47:11.554ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b") failed: Property not found. retry in 1.091483399s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
29612024-07-19T17:47:11.635ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52") failed: Property not found. retry in 1.250420017s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
29622024-07-19T17:47:11.674ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9") failed: Property not found. retry in 780.054774ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
29632024-07-19T17:47:11.699ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec") failed: Property not found. retry in 1.036939958s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
29642024-07-19T17:47:11.720ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b") failed: Property not found. retry in 754.801684ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
29652024-07-19T17:47:11.727ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7") failed: Property not found. retry in 884.913738ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
29662024-07-19T17:47:11.880ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603") failed: Property not found. retry in 1.181368032s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
29672024-07-19T17:47:12.004ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1") failed: Property not found. retry in 1.234596674s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
29682024-07-19T17:47:12.013ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17") failed: Property not found. retry in 1.239400401s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
29692024-07-19T17:47:12.071ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1") failed: Property not found. retry in 1.0258881s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
29702024-07-19T17:47:12.096ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660") failed: Property not found. retry in 1.172807247s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
29712024-07-19T17:47:12.124ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1") failed: Property not found. retry in 946.939416ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
29722024-07-19T17:47:12.203ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957") failed: Property not found. retry in 568.08243ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
29732024-07-19T17:47:12.219ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd") failed: Property not found. retry in 935.265526ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
29742024-07-19T17:47:12.329ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1") failed: Property not found. retry in 1.123041523s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
29752024-07-19T17:47:12.368ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203") failed: Property not found. retry in 742.485142ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
29762024-07-19T17:47:12.420ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb") failed: Property not found. retry in 869.466075ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
29772024-07-19T17:47:12.464ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9") failed: Property not found. retry in 621.783326ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
29782024-07-19T17:47:12.484ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b") failed: Property not found. retry in 1.327329272s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
29792024-07-19T17:47:12.608ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f") failed: Property not found. retry in 556.855942ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
29802024-07-19T17:47:12.619ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7") failed: Property not found. retry in 1.445001468s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
29812024-07-19T17:47:12.655ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b") failed: Property not found. retry in 1.107549634s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
29822024-07-19T17:47:12.746ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec") failed: Property not found. retry in 564.820143ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
29832024-07-19T17:47:12.762ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284") failed: Property not found. retry in 896.92475ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
29842024-07-19T17:47:12.780ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957") failed: Property not found. retry in 555.200717ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
29852024-07-19T17:47:12.894ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52") failed: Property not found. retry in 723.947718ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
29862024-07-19T17:47:13.073ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603") failed: Property not found. retry in 781.233115ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
29872024-07-19T17:47:13.081ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1") failed: Property not found. retry in 641.632247ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
29882024-07-19T17:47:13.096ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9") failed: Property not found. retry in 1.4921021s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
29892024-07-19T17:47:13.107ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1") failed: Property not found. retry in 782.654553ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
29902024-07-19T17:47:13.120ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203") failed: Property not found. retry in 1.197434739s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
29912024-07-19T17:47:13.163ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd") failed: Property not found. retry in 573.543571ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
29922024-07-19T17:47:13.175ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f") failed: Property not found. retry in 507.761374ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
29932024-07-19T17:47:13.247ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1") failed: Property not found. retry in 1.287111127s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
29942024-07-19T17:47:13.262ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17") failed: Property not found. retry in 1.403151212s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
29952024-07-19T17:47:13.278ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660") failed: Property not found. retry in 916.980655ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
29962024-07-19T17:47:13.299ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb") failed: Property not found. retry in 843.740343ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
29972024-07-19T17:47:13.321ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec") failed: Property not found. retry in 744.61498ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
29982024-07-19T17:47:13.345ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957") failed: Property not found. retry in 759.647302ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
29992024-07-19T17:47:13.461ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1") failed: Property not found. retry in 643.620871ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
30002024-07-19T17:47:13.628ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52") failed: Property not found. retry in 770.758975ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
30012024-07-19T17:47:13.667ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284") failed: Property not found. retry in 504.184771ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
30022024-07-19T17:47:13.692ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f") failed: Property not found. retry in 1.180343886s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
30032024-07-19T17:47:13.732ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1") failed: Property not found. retry in 1.265324031s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
30042024-07-19T17:47:13.746ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd") failed: Property not found. retry in 591.569499ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
30052024-07-19T17:47:13.773ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b") failed: Property not found. retry in 667.969785ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
30062024-07-19T17:47:13.820ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b") failed: Property not found. retry in 1.276313819s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
30072024-07-19T17:47:13.938ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1") failed: Property not found. retry in 1.372615146s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
30082024-07-19T17:47:14.073ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7") failed: Property not found. retry in 1.499873103s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
30092024-07-19T17:47:14.082ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec") failed: Property not found. retry in 780.072551ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
30102024-07-19T17:47:14.125ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1") failed: Property not found. retry in 1.469253291s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
30112024-07-19T17:47:14.152ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb") failed: Property not found. retry in 1.269962949s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
30122024-07-19T17:47:14.182ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284") failed: Property not found. retry in 990.2464ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
30132024-07-19T17:47:14.210ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660") failed: Property not found. retry in 1.435290517s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
30142024-07-19T17:47:14.215ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:53433
30152024-07-19T17:47:14.284ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 69206 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:53433 req_id = 58e2bd4c-d4a3-487d-844c-cc68a8159e81 response_code = 200 uri = /baseboard
30162024-07-19T17:47:14.328ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203") failed: Property not found. retry in 1.221630825s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
30172024-07-19T17:47:14.347ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd") failed: Property not found. retry in 914.191553ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
30182024-07-19T17:47:14.408ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52") failed: Property not found. retry in 860.215895ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
30192024-07-19T17:47:14.450ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b") failed: Property not found. retry in 986.453614ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
30202024-07-19T17:47:14.545ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1") failed: Property not found. retry in 731.256232ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
30212024-07-19T17:47:14.597ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9") failed: Property not found. retry in 957.799811ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
30222024-07-19T17:47:14.885ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f") failed: Property not found. retry in 1.046983563s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
30232024-07-19T17:47:15.106ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b") failed: Property not found. retry in 1.049930299s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
30242024-07-19T17:47:15.270ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd") failed: Property not found. retry in 522.631368ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
30252024-07-19T17:47:15.278ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52") failed: Property not found. retry in 1.014942922s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
30262024-07-19T17:47:15.437ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb") failed: Property not found. retry in 722.923174ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
30272024-07-19T17:47:15.447ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_742e0330-77de-4723-a145-74751d41c73b") failed: Property not found. retry in 765.883383ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
30282024-07-19T17:47:15.630ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1") failed: Property not found. retry in 819.94498ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
30292024-07-19T17:47:15.944ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f") failed: Property not found. retry in 1.414666449s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
30302024-07-19T17:47:16.166ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b") failed: Property not found. retry in 1.253509313s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
30312024-07-19T17:47:17.498ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2980 zone = oxz_crucible_bd7f3d92-4497-48dd-9f51-160ce4d7f4d9
30322024-07-19T17:47:17.498ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2980 zone = oxz_crucible_c6d325c3-cddf-4528-8ba6-a083f6511284
30332024-07-19T17:47:17.498ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2980 zone = oxz_crucible_c30b7bc1-fd10-4942-8055-ada48d2e3957
30342024-07-19T17:47:17.498ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2980 zone = oxz_external_dns_1eb90f9c-ba99-428e-885b-5fa2df7ae2f1
30352024-07-19T17:47:17.498ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2980 zone = oxz_nexus_742e0330-77de-4723-a145-74751d41c73b
30362024-07-19T17:47:17.498ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2980 zone = oxz_crucible_pantry_3286a4ca-fffd-4ed0-9c21-d528e1d23603
30372024-07-19T17:47:17.498ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2980 zone = oxz_crucible_6fa8dbd2-a741-440b-98a2-b60bf983cdc7
30382024-07-19T17:47:17.498ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2980 zone = oxz_clickhouse_2212adda-2fcf-4d59-888c-2223a4fe3f6b
30392024-07-19T17:47:17.498ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2980 zone = oxz_crucible_1bd16d4a-6b16-4df4-b218-9e6f3ad1d9fb
30402024-07-19T17:47:17.498ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2980 zone = oxz_crucible_139ec2b6-5f74-4c07-9d4b-6e2f8dbeab17
30412024-07-19T17:47:17.498ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2980 zone = oxz_nexus_87d65a87-e1e5-43e4-9cba-b4afd5e9ac52
30422024-07-19T17:47:17.498ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2980 zone = oxz_nexus_6088410e-a8fc-4d33-b593-6385e863291f
30432024-07-19T17:47:17.498ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2980 zone = oxz_oximeter_8d644923-5967-4351-a1ad-fe8a938f33e1
30442024-07-19T17:47:17.498ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2980 zone = oxz_crucible_pantry_64cdbe7d-ce11-4095-b7f7-bf51bfb3f7ec
30452024-07-19T17:47:17.498ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2980 zone = oxz_crucible_pantry_740bd3e0-b30c-4033-968a-5451c8d5adcd
30462024-07-19T17:47:17.498ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2980 zone = oxz_crucible_0b1925c8-3ae5-4372-a3de-213551574fe1
30472024-07-19T17:47:17.498ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2980 zone = oxz_crucible_663bd1f7-7a29-484c-bfce-d312646b1660
30482024-07-19T17:47:17.498ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2980 zone = oxz_crucible_605558ae-0f1e-47a7-b88c-f397548d4ea1
30492024-07-19T17:47:17.498ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2980 zone = oxz_external_dns_09fdc6fd-8297-436e-903d-b18710599203
30502024-07-19T17:47:17.513ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
30512024-07-19T17:47:17.514ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
30522024-07-19T17:47:17.529ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 29065749 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:37752 req_id = a8598c33-c3b0-4fed-943d-33cf06836681 response_code = 204 uri = /omicron-zones
30532024-07-19T17:47:17.618ZINFOSledAgent (RSS): Finished setting up services file = sled-agent/src/rack_setup/service.rs:1244
30542024-07-19T17:47:17.618ZINFOSledAgent (RSS): Writing ledger to /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
30552024-07-19T17:47:17.619ZINFOSledAgent (RSS): Writing ledger to /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
30562024-07-19T17:47:17.623ZINFOSledAgent (RSS): Handing off control to Nexus file = sled-agent/src/rack_setup/service.rs:681
30572024-07-19T17:47:17.647ZINFOSledAgent (RSS): Nexus address: [fd00:1122:3344:101::b]:12221 file = sled-agent/src/rack_setup/service.rs:694
30582024-07-19T17:47:17.761ZINFOSledAgent (RSS): rack_network_config: RackNetworkConfigV2 { bfd: [], bgp: [], infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV2 { addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.1.174, width: 24, }, ), vlan_id: None, }, ], autoneg: false, bgp_peers: [], port: "qsfp0", routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.1.100, vlan_id: None, }, ], switch: Switch0, uplink_port_fec: None, uplink_port_speed: Speed40G, }, ], rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, } file = sled-agent/src/rack_setup/service.rs:825
30592024-07-19T17:47:20.876ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 89.091825153s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error trying to connect: tcp connect error: Connection refused (os error 146)" file = oximeter/producer/src/lib.rs:424 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
30602024-07-19T17:47:20.876ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/0ac29522-8a9f-4432-bdfe-a8adde17b3b5/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
30612024-07-19T17:47:20.977ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/0ac29522-8a9f-4432-bdfe-a8adde17b3b5/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
30622024-07-19T17:47:21.100ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/0ac29522-8a9f-4432-bdfe-a8adde17b3b5/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
30632024-07-19T17:47:21.277ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/0ac29522-8a9f-4432-bdfe-a8adde17b3b5/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
30642024-07-19T17:47:21.377ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/0ac29522-8a9f-4432-bdfe-a8adde17b3b5/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
30652024-07-19T17:47:21.655ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/0ac29522-8a9f-4432-bdfe-a8adde17b3b5/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
30662024-07-19T17:47:21.805ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/0ac29522-8a9f-4432-bdfe-a8adde17b3b5/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
30672024-07-19T17:47:22.065ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/0ac29522-8a9f-4432-bdfe-a8adde17b3b5/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
30682024-07-19T17:47:22.315ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/0ac29522-8a9f-4432-bdfe-a8adde17b3b5/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
30692024-07-19T17:47:22.841ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/0ac29522-8a9f-4432-bdfe-a8adde17b3b5/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
30702024-07-19T17:47:23.192ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/0ac29522-8a9f-4432-bdfe-a8adde17b3b5/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
30712024-07-19T17:47:23.685ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/0ac29522-8a9f-4432-bdfe-a8adde17b3b5/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
30722024-07-19T17:47:24.555ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/0ac29522-8a9f-4432-bdfe-a8adde17b3b5/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
30732024-07-19T17:47:25.340ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/0ac29522-8a9f-4432-bdfe-a8adde17b3b5/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
30742024-07-19T17:47:26.018ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/0ac29522-8a9f-4432-bdfe-a8adde17b3b5/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
30752024-07-19T17:47:27.349ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/0ac29522-8a9f-4432-bdfe-a8adde17b3b5/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
30762024-07-19T17:47:29.298ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/0ac29522-8a9f-4432-bdfe-a8adde17b3b5/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
30772024-07-19T17:47:31.185ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 403 Forbidden; headers: {"content-type": "application/json", "x-request-id": "eaadb9e4-0410-44d9-9ac4-6143382572d6", "content-length": "113", "date": "Fri, 19 Jul 2024 17:47:30 GMT"}; value: Error { error_code: Some("Forbidden"), message: "Forbidden", request_id: "eaadb9e4-0410-44d9-9ac4-6143382572d6" } file = sled-agent/src/rack_setup/service.rs:887
30782024-07-19T17:47:34.324ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "ba92bfcd-3ffe-4a1a-bf14-9418c6f5471b", "content-length": "173", "date": "Fri, 19 Jul 2024 17:47:33 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"0c1108fb-cc91-4535-adf7-846d50ef8716\"", request_id: "ba92bfcd-3ffe-4a1a-bf14-9418c6f5471b" } file = sled-agent/src/rack_setup/service.rs:887
30792024-07-19T17:47:36.071ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "73264092-d8e0-426e-b2f5-385261646d70", "content-length": "173", "date": "Fri, 19 Jul 2024 17:47:36 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"0c1108fb-cc91-4535-adf7-846d50ef8716\"", request_id: "73264092-d8e0-426e-b2f5-385261646d70" } file = sled-agent/src/rack_setup/service.rs:887
30802024-07-19T17:47:39.855ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "0f4ca73b-94f3-4cfa-9dc7-274c3c32fc2e", "content-length": "173", "date": "Fri, 19 Jul 2024 17:47:39 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"0c1108fb-cc91-4535-adf7-846d50ef8716\"", request_id: "0f4ca73b-94f3-4cfa-9dc7-274c3c32fc2e" } file = sled-agent/src/rack_setup/service.rs:887
30812024-07-19T17:47:44.322ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:51922
30822024-07-19T17:47:44.322ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 38 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:51922 req_id = 884e6751-75c0-499f-8b11-f6d0f242f037 response_code = 200 uri = /baseboard
30832024-07-19T17:47:44.343ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "633313ed-9e18-45ad-a875-7ec65f7b5549", "content-length": "173", "date": "Fri, 19 Jul 2024 17:47:44 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"0c1108fb-cc91-4535-adf7-846d50ef8716\"", request_id: "633313ed-9e18-45ad-a875-7ec65f7b5549" } file = sled-agent/src/rack_setup/service.rs:887
30842024-07-19T17:47:51.379ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "9572ae4d-01cb-4300-862c-0851bce7d8e8", "content-length": "173", "date": "Fri, 19 Jul 2024 17:47:51 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"0c1108fb-cc91-4535-adf7-846d50ef8716\"", request_id: "9572ae4d-01cb-4300-862c-0851bce7d8e8" } file = sled-agent/src/rack_setup/service.rs:887
30852024-07-19T17:47:55.307ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "4219d8e3-8b7c-4595-89b5-352da8999381", "content-length": "173", "date": "Fri, 19 Jul 2024 17:47:55 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"0c1108fb-cc91-4535-adf7-846d50ef8716\"", request_id: "4219d8e3-8b7c-4595-89b5-352da8999381" } file = sled-agent/src/rack_setup/service.rs:887
30862024-07-19T17:47:58.291ZINFOSledAgent (NexusNotifierTask): Successfully put SledAgentInfo to nexus file = sled-agent/src/nexus.rs:592 new_generation = 1 old_generation = 1 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
30872024-07-19T17:47:58.421ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:34400
30882024-07-19T17:47:58.425ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
30892024-07-19T17:47:58.512ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 89540 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:34400 req_id = 0b4fae87-f976-4130-80d8-34c2cc51170e response_code = 200 uri = /inventory
30902024-07-19T17:47:58.514ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
30912024-07-19T17:47:58.514ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/omicron-zones.json file = common/src/ledger.rs:177
30922024-07-19T17:47:58.520ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/omicron-zones.json file = common/src/ledger.rs:177
30932024-07-19T17:47:58.521ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 7234 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:34400 req_id = 8382a9b4-52e2-4115-87e2-011395352603 response_code = 200 uri = /omicron-zones
30942024-07-19T17:48:01.774ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37545
30952024-07-19T17:48:01.774ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 129 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:37545 req_id = 9ff43996-de9a-419d-b71b-4b0cad3a4bf9 response_code = 200 uri = /vpc-routes
30962024-07-19T17:48:01.859ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 8923 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:37545 req_id = 094dc056-ea38-4a54-ae31-60f487ec1840 response_code = 204 uri = /vpc-routes
30972024-07-19T17:48:01.865ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37450
30982024-07-19T17:48:01.865ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 280 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:37450 req_id = 335c306a-3a60-4ef0-929e-2ac8f14b5f02 response_code = 200 uri = /v2p
30992024-07-19T17:48:01.946ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:39199
31002024-07-19T17:48:01.946ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 591 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:39199 req_id = 25cec175-cb10-4ab6-b289-3f365d044509 response_code = 200 uri = /vpc-routes
31012024-07-19T17:48:01.950ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:37624
31022024-07-19T17:48:01.950ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 100 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:37624 req_id = cd460589-0067-4a26-851d-fcb93bcc228f response_code = 200 uri = /v2p
31032024-07-19T17:48:01.953ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 72 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:39199 req_id = 713ac1a5-9369-4faa-b386-bd08de04d244 response_code = 204 uri = /vpc-routes
31042024-07-19T17:48:02.583ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:65207
31052024-07-19T17:48:02.583ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 217 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:65207 req_id = a4fedbcb-3f61-4b68-91fd-c9b3f9aa6be7 response_code = 200 uri = /v2p
31062024-07-19T17:48:02.656ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:51787
31072024-07-19T17:48:02.656ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 68 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:51787 req_id = bc98ed30-392c-4c16-9062-499f2863bfad response_code = 200 uri = /vpc-routes
31082024-07-19T17:48:02.662ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 130 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:51787 req_id = 74f32ab7-e650-46e3-accb-5531d1063799 response_code = 204 uri = /vpc-routes
31092024-07-19T17:48:05.122ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:48065
31102024-07-19T17:48:05.125ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:673 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 005b2740-84d5-473e-ae59-aa79fd2fe5b6, kind: Service { id: 09fdc6fd-8297-436e-903d-b18710599203 }, name: Name("external-dns-09fdc6fd-8297-436e-903d-b18710599203"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 6])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 59731bd7-ffc9-493f-a33c-a1cf5ba2df30, kind: Service { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1 }, name: Name("external-dns-1eb90f9c-ba99-428e-885b-5fa2df7ae2f1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 32])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: c6edf68c-f558-4f51-9aed-6eb91fb3ad51, kind: Service { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52 }, name: Name("nexus-87d65a87-e1e5-43e4-9cba-b4afd5e9ac52"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 117])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e6898038-bf11-41c8-b971-6ff3eed6d38b, kind: Service { id: 6088410e-a8fc-4d33-b593-6385e863291f }, name: Name("nexus-6088410e-a8fc-4d33-b593-6385e863291f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 27])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: ffe6d41d-1349-42b2-a887-49d734eef422, kind: Service { id: 742e0330-77de-4723-a145-74751d41c73b }, name: Name("nexus-742e0330-77de-4723-a145-74751d41c73b"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 199, 184])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
31112024-07-19T17:48:05.126ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
31122024-07-19T17:48:05.127ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
31132024-07-19T17:48:05.127ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte0" rules = []
31142024-07-19T17:48:05.127ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
31152024-07-19T17:48:05.127ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
31162024-07-19T17:48:05.127ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
31172024-07-19T17:48:05.127ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 5275 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:48065 req_id = d3a78ea5-b9af-475e-af09-bd0c607630eb response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
31182024-07-19T17:48:05.129ZINFOSledAgent (RSS): Handoff to Nexus is complete file = sled-agent/src/rack_setup/service.rs:897
31192024-07-19T17:48:05.178ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63841
31202024-07-19T17:48:05.179ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:673 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 005b2740-84d5-473e-ae59-aa79fd2fe5b6, kind: Service { id: 09fdc6fd-8297-436e-903d-b18710599203 }, name: Name("external-dns-09fdc6fd-8297-436e-903d-b18710599203"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 6])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 59731bd7-ffc9-493f-a33c-a1cf5ba2df30, kind: Service { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1 }, name: Name("external-dns-1eb90f9c-ba99-428e-885b-5fa2df7ae2f1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 32])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: c6edf68c-f558-4f51-9aed-6eb91fb3ad51, kind: Service { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52 }, name: Name("nexus-87d65a87-e1e5-43e4-9cba-b4afd5e9ac52"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 117])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e6898038-bf11-41c8-b971-6ff3eed6d38b, kind: Service { id: 6088410e-a8fc-4d33-b593-6385e863291f }, name: Name("nexus-6088410e-a8fc-4d33-b593-6385e863291f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 27])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: ffe6d41d-1349-42b2-a887-49d734eef422, kind: Service { id: 742e0330-77de-4723-a145-74751d41c73b }, name: Name("nexus-742e0330-77de-4723-a145-74751d41c73b"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 199, 184])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
31212024-07-19T17:48:05.179ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
31222024-07-19T17:48:05.179ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
31232024-07-19T17:48:05.179ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte0" rules = []
31242024-07-19T17:48:05.179ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
31252024-07-19T17:48:05.179ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
31262024-07-19T17:48:05.179ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
31272024-07-19T17:48:05.179ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 350 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:63841 req_id = c1a547f0-1063-4d70-8210-89a152e20d30 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
31282024-07-19T17:48:05.180ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44708
31292024-07-19T17:48:05.180ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:673 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 005b2740-84d5-473e-ae59-aa79fd2fe5b6, kind: Service { id: 09fdc6fd-8297-436e-903d-b18710599203 }, name: Name("external-dns-09fdc6fd-8297-436e-903d-b18710599203"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 6])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 59731bd7-ffc9-493f-a33c-a1cf5ba2df30, kind: Service { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1 }, name: Name("external-dns-1eb90f9c-ba99-428e-885b-5fa2df7ae2f1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 32])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: c6edf68c-f558-4f51-9aed-6eb91fb3ad51, kind: Service { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52 }, name: Name("nexus-87d65a87-e1e5-43e4-9cba-b4afd5e9ac52"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 117])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e6898038-bf11-41c8-b971-6ff3eed6d38b, kind: Service { id: 6088410e-a8fc-4d33-b593-6385e863291f }, name: Name("nexus-6088410e-a8fc-4d33-b593-6385e863291f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 27])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: ffe6d41d-1349-42b2-a887-49d734eef422, kind: Service { id: 742e0330-77de-4723-a145-74751d41c73b }, name: Name("nexus-742e0330-77de-4723-a145-74751d41c73b"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 199, 184])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
31302024-07-19T17:48:05.180ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
31312024-07-19T17:48:05.180ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
31322024-07-19T17:48:05.180ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte0" rules = []
31332024-07-19T17:48:05.180ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
31342024-07-19T17:48:05.180ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
31352024-07-19T17:48:05.180ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
31362024-07-19T17:48:05.180ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 566 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:44708 req_id = a2a37b9e-3ae5-472a-8da1-c1a7accb085e response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
31372024-07-19T17:48:05.208ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47926
31382024-07-19T17:48:05.209ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:673 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 005b2740-84d5-473e-ae59-aa79fd2fe5b6, kind: Service { id: 09fdc6fd-8297-436e-903d-b18710599203 }, name: Name("external-dns-09fdc6fd-8297-436e-903d-b18710599203"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 6])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 59731bd7-ffc9-493f-a33c-a1cf5ba2df30, kind: Service { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1 }, name: Name("external-dns-1eb90f9c-ba99-428e-885b-5fa2df7ae2f1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 32])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: c6edf68c-f558-4f51-9aed-6eb91fb3ad51, kind: Service { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52 }, name: Name("nexus-87d65a87-e1e5-43e4-9cba-b4afd5e9ac52"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 117])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e6898038-bf11-41c8-b971-6ff3eed6d38b, kind: Service { id: 6088410e-a8fc-4d33-b593-6385e863291f }, name: Name("nexus-6088410e-a8fc-4d33-b593-6385e863291f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 27])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: ffe6d41d-1349-42b2-a887-49d734eef422, kind: Service { id: 742e0330-77de-4723-a145-74751d41c73b }, name: Name("nexus-742e0330-77de-4723-a145-74751d41c73b"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 199, 184])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
31392024-07-19T17:48:05.209ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
31402024-07-19T17:48:05.209ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
31412024-07-19T17:48:05.209ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte0" rules = []
31422024-07-19T17:48:05.209ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
31432024-07-19T17:48:05.209ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
31442024-07-19T17:48:05.209ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
31452024-07-19T17:48:05.209ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 250 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:47926 req_id = c66c2e62-678e-4af5-9cd4-1d507baec81d response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
31462024-07-19T17:48:05.381ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:57723
31472024-07-19T17:48:05.382ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
31482024-07-19T17:48:05.473ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 92076 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:57723 req_id = a429c85c-bfa5-4e86-9671-50475d8929d6 response_code = 200 uri = /inventory
31492024-07-19T17:48:05.473ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
31502024-07-19T17:48:05.473ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/config/omicron-zones.json file = common/src/ledger.rs:177
31512024-07-19T17:48:05.474ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/config/omicron-zones.json file = common/src/ledger.rs:177
31522024-07-19T17:48:05.474ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 1421 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:57723 req_id = c56d0820-50a1-4d26-9b40-9c259392eac1 response_code = 200 uri = /omicron-zones
31532024-07-19T17:48:10.155ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:39577
31542024-07-19T17:48:10.155ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 121 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:39577 req_id = 289f51d1-3b2e-4356-bf0b-ac0feb39de18 response_code = 200 uri = /vpc-routes
31552024-07-19T17:48:10.161ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 131 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:39577 req_id = 456702e3-7d54-4f01-87d4-fb65ea6c466f response_code = 204 uri = /vpc-routes
31562024-07-19T17:48:14.361ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:40047
31572024-07-19T17:48:14.361ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 629 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:40047 req_id = adfbf536-ad08-4a95-945f-53152c55b499 response_code = 200 uri = /baseboard
31582024-07-19T17:48:31.834ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59044
31592024-07-19T17:48:31.834ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 155 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:59044 req_id = 909343c7-f770-4b7b-8da3-2407d8424e80 response_code = 200 uri = /vpc-routes
31602024-07-19T17:48:31.842ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 138 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:59044 req_id = dfe045f6-0ce5-4a0f-86b5-0ee47bf579ed response_code = 204 uri = /vpc-routes
31612024-07-19T17:48:32.081ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:34217
31622024-07-19T17:48:32.082ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 746 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:34217 req_id = 73f4c82b-57d4-474c-bd61-388cf171fe3c response_code = 200 uri = /vpc-routes
31632024-07-19T17:48:32.083ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61018
31642024-07-19T17:48:32.083ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 135 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:61018 req_id = 5de6a38f-204f-466a-aadc-9ab4825400c5 response_code = 200 uri = /v2p
31652024-07-19T17:48:32.089ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 84 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:34217 req_id = 756db38c-cee9-44a5-8763-8b9d1644d26c response_code = 204 uri = /vpc-routes
31662024-07-19T17:48:32.148ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:36558
31672024-07-19T17:48:32.149ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 742 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:36558 req_id = 14434cfc-2264-485f-bcdf-a034a2b3c066 response_code = 200 uri = /v2p
31682024-07-19T17:48:32.333ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:46988
31692024-07-19T17:48:32.418ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 84707 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:46988 req_id = ffd15340-83f0-4393-83c7-243114cf2b08 response_code = 204 uri = /switch-ports
31702024-07-19T17:48:32.419ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 1232 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:46988 req_id = 4d3d5da6-3b91-4b18-a9f8-51393a155d35 response_code = 200 uri = /network-bootstore-config
31712024-07-19T17:48:32.442ZINFOSledAgent (bootstore): Attempting to update network config with generation: 2, current_generation: 1 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-buskin-i86pc
31722024-07-19T17:48:32.442ZINFOSledAgent (bootstore): Writing ledger to /pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
31732024-07-19T17:48:32.444ZINFOSledAgent (bootstore): Writing ledger to /pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
31742024-07-19T17:48:32.446ZINFOSledAgent (bootstore): Broadcasting network config with generation 2 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-buskin-i86pc
31752024-07-19T17:48:32.446ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 6921 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:46988 req_id = 31309138-162e-4d8f-9dce-7e4b65a1a34a response_code = 204 uri = /network-bootstore-config
31762024-07-19T17:48:32.447ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:46620
31772024-07-19T17:48:32.508ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 60548 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:46620 req_id = c49be600-41d1-40c9-abd2-7a8ea640fdd2 response_code = 204 uri = /switch-ports
31782024-07-19T17:48:32.511ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 846 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:46620 req_id = b3ca0fbe-235c-4793-b7ec-8c853d12b653 response_code = 200 uri = /network-bootstore-config
31792024-07-19T17:48:32.930ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:35675
31802024-07-19T17:48:32.931ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 926 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:35675 req_id = edac0aec-e2ef-48a9-870c-16e555bcc5e5 response_code = 200 uri = /v2p
31812024-07-19T17:48:32.948ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:65195
31822024-07-19T17:48:32.948ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 53 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:65195 req_id = b05ec78b-98f8-413e-bd1a-830e53673e5c response_code = 200 uri = /vpc-routes
31832024-07-19T17:48:32.956ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 98 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:65195 req_id = 12ec588f-525e-4ec3-8c6f-bfc7b338c647 response_code = 204 uri = /vpc-routes
31842024-07-19T17:48:33.043ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:45639
31852024-07-19T17:48:33.093ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 50147 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:45639 req_id = 888e05e2-1c71-46c4-9674-66454a1c6ab4 response_code = 204 uri = /switch-ports
31862024-07-19T17:48:33.096ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 815 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:45639 req_id = 4e226918-16ff-438d-b1b0-4bcbf6fb1f95 response_code = 200 uri = /network-bootstore-config
31872024-07-19T17:48:44.395ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:45020
31882024-07-19T17:48:44.395ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 470 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:45020 req_id = 0d2e7e58-41a9-42d9-b79e-81ea0186e7cf response_code = 200 uri = /baseboard
31892024-07-19T17:49:01.851ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:64089
31902024-07-19T17:49:01.851ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 209 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:64089 req_id = b63f95fd-0e0b-42dd-a476-40f1ed968db9 response_code = 200 uri = /v2p
31912024-07-19T17:49:01.895ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:50650
31922024-07-19T17:49:01.895ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 70 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:50650 req_id = 1d8aaebb-d230-47cb-be51-35ca2d76f7a8 response_code = 200 uri = /vpc-routes
31932024-07-19T17:49:01.903ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 113 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:50650 req_id = 6ba68314-4fc6-4848-bc0b-57a5ca47f35b response_code = 204 uri = /vpc-routes
31942024-07-19T17:49:01.951ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:60800
31952024-07-19T17:49:01.952ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 759 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:60800 req_id = 959407d0-ee69-48a7-98ed-ce4add78016e response_code = 200 uri = /v2p
31962024-07-19T17:49:01.955ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:55661
31972024-07-19T17:49:01.955ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 24 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:55661 req_id = ce73ea46-82e5-4d64-a2d9-305cece0dcf0 response_code = 200 uri = /vpc-routes
31982024-07-19T17:49:01.962ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 55 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:55661 req_id = 31ab91aa-609e-474d-85ef-154a1d0bbf9e response_code = 204 uri = /vpc-routes
31992024-07-19T17:49:01.988ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:60361
32002024-07-19T17:49:02.035ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 47180 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:60361 req_id = 85b8e6a8-c79d-4baf-b6e9-b20a6aea639b response_code = 204 uri = /switch-ports
32012024-07-19T17:49:02.035ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 86 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:60361 req_id = e5bf7201-f353-45fe-8d7d-d0561cb168d1 response_code = 200 uri = /network-bootstore-config
32022024-07-19T17:49:02.065ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:33431
32032024-07-19T17:49:02.111ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 45515 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:33431 req_id = a58d5e5a-451a-482e-9095-39754e840288 response_code = 204 uri = /switch-ports
32042024-07-19T17:49:02.111ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 103 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:33431 req_id = 285042f9-c52d-4302-8370-c2b38452d6a9 response_code = 200 uri = /network-bootstore-config
32052024-07-19T17:49:02.605ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:33411
32062024-07-19T17:49:02.606ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 219 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:33411 req_id = e5df0bcf-d7b7-4c21-87cf-708ec5b4c741 response_code = 200 uri = /v2p
32072024-07-19T17:49:02.738ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:49041
32082024-07-19T17:49:02.738ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 81 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:49041 req_id = fad4eac7-16fd-4489-bb45-c2ebcf6f7abf response_code = 200 uri = /vpc-routes
32092024-07-19T17:49:02.745ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 101 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:49041 req_id = 056da02f-78d2-4d5c-82c3-8c11767f23eb response_code = 204 uri = /vpc-routes
32102024-07-19T17:49:02.828ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:65512
32112024-07-19T17:49:02.875ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 47030 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:65512 req_id = 7fc059e9-806e-4169-825c-58bd30e33944 response_code = 204 uri = /switch-ports
32122024-07-19T17:49:02.875ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 106 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:65512 req_id = a7c9141c-ed5a-4591-8c5e-c9ac95916210 response_code = 200 uri = /network-bootstore-config
32132024-07-19T17:49:14.428ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:40930
32142024-07-19T17:49:14.428ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 95 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:40930 req_id = 8fbecb73-80e7-4cf9-841a-5843919d6ab4 response_code = 200 uri = /baseboard
32152024-07-19T17:49:20.115ZINFOSledAgent (dropshot): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:56984 remote_addr = [fd00:1122:3344:101::d]:57512 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716
32162024-07-19T17:49:20.127ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 9058 local_addr = [fd00:1122:3344:101::1]:56984 method = GET remote_addr = [fd00:1122:3344:101::d]:57512 req_id = dfbeeb30-5a37-4cb3-8118-aa134b5aa293 response_code = 200 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716 uri = /0c1108fb-cc91-4535-adf7-846d50ef8716
32172024-07-19T17:49:31.516ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
32182024-07-19T17:49:31.928ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:51195
32192024-07-19T17:49:31.928ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 71 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:51195 req_id = 76e7a104-db4c-4a51-bdf9-28664329e323 response_code = 200 uri = /vpc-routes
32202024-07-19T17:49:31.936ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 117 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:51195 req_id = 84fc0552-3919-4d7d-85b0-b6f9b6c769f1 response_code = 204 uri = /vpc-routes
32212024-07-19T17:49:31.963ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:34046
32222024-07-19T17:49:31.963ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 116 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:34046 req_id = c6bbe0bf-b338-40c9-a230-f8d9b6e00242 response_code = 200 uri = /v2p
32232024-07-19T17:49:31.969ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36894
32242024-07-19T17:49:31.970ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 766 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:36894 req_id = a355ef69-f7df-4664-b9d7-43c8d11b17c4 response_code = 200 uri = /v2p
32252024-07-19T17:49:32.050ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42701
32262024-07-19T17:49:32.050ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 35 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:42701 req_id = 0440cab7-639c-463e-bb5d-f64ba8123093 response_code = 200 uri = /vpc-routes
32272024-07-19T17:49:32.059ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 87 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:42701 req_id = 37615928-4f6b-478c-b1f4-04ddd305ab93 response_code = 204 uri = /vpc-routes
32282024-07-19T17:49:32.222ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:46455
32292024-07-19T17:49:32.273ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 50292 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:46455 req_id = 92263a3f-12a8-40dc-a446-024c54efa8cd response_code = 204 uri = /switch-ports
32302024-07-19T17:49:32.273ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:60784
32312024-07-19T17:49:32.276ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 3219 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:46455 req_id = bd2590ce-9754-4a86-8e07-e4b9acf37dae response_code = 200 uri = /network-bootstore-config
32322024-07-19T17:49:32.320ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 47389 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:60784 req_id = 6e34021c-50a8-48f1-96aa-e0668f303d1f response_code = 204 uri = /switch-ports
32332024-07-19T17:49:32.321ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 801 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:60784 req_id = d8f60fb5-a2bf-490b-842f-71c142445ef4 response_code = 200 uri = /network-bootstore-config
32342024-07-19T17:49:32.618ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:54325
32352024-07-19T17:49:32.618ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 66 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:54325 req_id = dffaffff-66f1-487e-ae43-36cb1aadf188 response_code = 200 uri = /vpc-routes
32362024-07-19T17:49:32.627ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 115 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:54325 req_id = 7d85a663-598f-4d5e-a6cc-862aab1fcaae response_code = 204 uri = /vpc-routes
32372024-07-19T17:49:32.718ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:63715
32382024-07-19T17:49:32.718ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 139 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:63715 req_id = a7eeb8fb-bbaa-499b-ac30-4bc7d618b928 response_code = 200 uri = /v2p
32392024-07-19T17:49:32.935ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:49458
32402024-07-19T17:49:32.983ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 48159 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:49458 req_id = 4b80fa45-1c26-44e1-b8e1-58a244d3cc3a response_code = 204 uri = /switch-ports
32412024-07-19T17:49:32.983ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 77 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:49458 req_id = f251e753-1018-4a45-b515-74fc7a38e367 response_code = 200 uri = /network-bootstore-config
32422024-07-19T17:49:44.462ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:53010
32432024-07-19T17:49:44.462ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 118 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:53010 req_id = 231d75ad-4916-4cc6-bda9-92b6f1206af1 response_code = 200 uri = /baseboard
32442024-07-19T17:49:50.114ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 1093 local_addr = [fd00:1122:3344:101::1]:56984 method = GET remote_addr = [fd00:1122:3344:101::d]:57512 req_id = 43ca6d3c-15a9-4a2d-aeaf-a7d44c1b89b8 response_code = 200 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716 uri = /0c1108fb-cc91-4535-adf7-846d50ef8716
32452024-07-19T17:50:01.814ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:35483
32462024-07-19T17:50:01.815ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 925 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:35483 req_id = a5287d41-3278-4f42-aa53-38c37951d6b0 response_code = 200 uri = /vpc-routes
32472024-07-19T17:50:01.815ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:60930
32482024-07-19T17:50:01.815ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 40 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:60930 req_id = 4ddfd068-9dd5-45c1-815b-7e95b9cbda03 response_code = 200 uri = /vpc-routes
32492024-07-19T17:50:01.823ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 124 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:35483 req_id = f2cc2ee5-5639-43b8-848f-2f53d37d85ba response_code = 204 uri = /vpc-routes
32502024-07-19T17:50:01.823ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 43 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:60930 req_id = 0c58c2f3-17fc-475f-9f4d-8afa742a202d response_code = 204 uri = /vpc-routes
32512024-07-19T17:50:01.846ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:38189
32522024-07-19T17:50:01.846ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 130 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:38189 req_id = deaa4e22-c86d-4a71-b658-447950c7dbb8 response_code = 200 uri = /v2p
32532024-07-19T17:50:01.851ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:55149
32542024-07-19T17:50:01.851ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 72 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:55149 req_id = 2b340d35-d029-4637-93dd-a98a7f1b9be6 response_code = 200 uri = /v2p
32552024-07-19T17:50:01.962ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:62960
32562024-07-19T17:50:02.012ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 49593 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:62960 req_id = db636788-1bc1-4d17-8f59-f90df23ef533 response_code = 204 uri = /switch-ports
32572024-07-19T17:50:02.013ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 721 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:62960 req_id = 50ec5868-b0c4-41cc-bc94-05c516376c79 response_code = 200 uri = /network-bootstore-config
32582024-07-19T17:50:02.050ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33239
32592024-07-19T17:50:02.096ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 45713 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:33239 req_id = 30a56b02-1ceb-4338-a263-9f31a03b77a5 response_code = 204 uri = /switch-ports
32602024-07-19T17:50:02.096ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:33239 req_id = 93ee3637-9ab1-4dec-b7f3-f5a401519cc9 response_code = 200 uri = /network-bootstore-config
32612024-07-19T17:50:02.588ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:60539
32622024-07-19T17:50:02.588ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 108 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:60539 req_id = 0ac852a9-a599-40b6-8709-a8534c90e087 response_code = 200 uri = /vpc-routes
32632024-07-19T17:50:02.596ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 120 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:60539 req_id = df69b1a0-0aea-4171-a443-fc984acaf770 response_code = 204 uri = /vpc-routes
32642024-07-19T17:50:02.687ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:62468
32652024-07-19T17:50:02.687ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 137 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:62468 req_id = 9f3db2dc-76af-461d-b381-c52df7b8b3c6 response_code = 200 uri = /v2p
32662024-07-19T17:50:02.792ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:59832
32672024-07-19T17:50:02.839ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 46825 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:59832 req_id = 1787f258-58e2-4b24-961b-3dda78ca857f response_code = 204 uri = /switch-ports
32682024-07-19T17:50:02.839ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 107 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:59832 req_id = fd8b856a-a566-447d-92e1-0d8c7b443e5f response_code = 200 uri = /network-bootstore-config
32692024-07-19T17:50:14.494ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:39981
32702024-07-19T17:50:14.495ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 549 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:39981 req_id = 9afb9039-2ad7-4752-bba8-a9a2d6126947 response_code = 200 uri = /baseboard
32712024-07-19T17:50:20.115ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 651 local_addr = [fd00:1122:3344:101::1]:56984 method = GET remote_addr = [fd00:1122:3344:101::d]:57512 req_id = af79bea8-7e77-4b9f-815a-3f8fc69af81f response_code = 200 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716 uri = /0c1108fb-cc91-4535-adf7-846d50ef8716
32722024-07-19T17:50:32.035ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:33959
32732024-07-19T17:50:32.035ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 180 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:33959 req_id = 91af9910-0c88-4da9-938f-6b7c3aacb1b6 response_code = 200 uri = /v2p
32742024-07-19T17:50:32.062ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:65308
32752024-07-19T17:50:32.063ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 1020 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:65308 req_id = 966c980b-3473-49be-8422-67671abccc06 response_code = 200 uri = /vpc-routes
32762024-07-19T17:50:32.072ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 119 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:65308 req_id = f0309646-b750-48d5-81ec-405042063c59 response_code = 204 uri = /vpc-routes
32772024-07-19T17:50:32.094ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33643
32782024-07-19T17:50:32.094ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 90 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:33643 req_id = 25f1da67-fb87-4839-a70c-a658f128407e response_code = 200 uri = /v2p
32792024-07-19T17:50:32.148ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34000
32802024-07-19T17:50:32.148ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 43 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:34000 req_id = dffe87eb-0e6c-492a-8a53-fc0b7803f251 response_code = 200 uri = /vpc-routes
32812024-07-19T17:50:32.155ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 79 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:34000 req_id = b663929f-1851-4e61-ab4e-e02a7d9791b8 response_code = 204 uri = /vpc-routes
32822024-07-19T17:50:32.320ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:38135
32832024-07-19T17:50:32.369ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 48294 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:38135 req_id = 1aee49a8-ac4c-40bd-80f6-887a0abf54fc response_code = 204 uri = /switch-ports
32842024-07-19T17:50:32.369ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:45921
32852024-07-19T17:50:32.415ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 46427 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:45921 req_id = c4d334af-975d-42a3-a1ff-e4fdff2068a4 response_code = 204 uri = /switch-ports
32862024-07-19T17:50:32.416ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 119 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:38135 req_id = 7e650900-f07b-4406-9116-7ecf2b91b078 response_code = 200 uri = /network-bootstore-config
32872024-07-19T17:50:32.417ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 884 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:45921 req_id = 0cc4c66e-9093-4fa4-9e6c-75d5ea51e94a response_code = 200 uri = /network-bootstore-config
32882024-07-19T17:50:32.680ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:55645
32892024-07-19T17:50:32.680ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 71 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:55645 req_id = 43af3bce-63f7-40b3-9145-e953f3fae620 response_code = 200 uri = /vpc-routes
32902024-07-19T17:50:32.688ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 891 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:55645 req_id = 0c60b911-63da-47a1-b426-99a65d1a1300 response_code = 204 uri = /vpc-routes
32912024-07-19T17:50:32.849ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:52710
32922024-07-19T17:50:32.849ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 124 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:52710 req_id = f37be28f-bb74-44ed-a084-fb2be1dea11d response_code = 200 uri = /v2p
32932024-07-19T17:50:33.060ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:59948
32942024-07-19T17:50:33.108ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 48329 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:59948 req_id = 62fb8251-4980-4852-b2e6-7455b4b7d526 response_code = 204 uri = /switch-ports
32952024-07-19T17:50:33.108ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 94 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:59948 req_id = 3621962d-8cb2-468e-afbe-301092d857d2 response_code = 200 uri = /network-bootstore-config
32962024-07-19T17:50:44.527ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:59582
32972024-07-19T17:50:44.528ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 622 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:59582 req_id = 31908f3c-16e8-47ab-a938-2c07bfa5c484 response_code = 200 uri = /baseboard
32982024-07-19T17:50:50.115ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 1049 local_addr = [fd00:1122:3344:101::1]:56984 method = GET remote_addr = [fd00:1122:3344:101::d]:57512 req_id = 9d4ce8c8-64cb-4396-ae3b-62cf68934958 response_code = 200 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716 uri = /0c1108fb-cc91-4535-adf7-846d50ef8716
32992024-07-19T17:51:01.823ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:48783
33002024-07-19T17:51:01.823ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 109 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:48783 req_id = 09946601-d7f4-4266-9437-e6f68b4b586e response_code = 200 uri = /vpc-routes
33012024-07-19T17:51:01.831ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 133 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:48783 req_id = 120580a5-d135-477e-82df-d6e677ef2fca response_code = 204 uri = /vpc-routes
33022024-07-19T17:51:01.905ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33274
33032024-07-19T17:51:01.906ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 837 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:33274 req_id = 307ae6b7-f4d1-43e3-887e-c7828fb10c67 response_code = 200 uri = /vpc-routes
33042024-07-19T17:51:01.914ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 98 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:33274 req_id = b7384a57-9bf3-4a30-add5-3aa4807c4c4c response_code = 204 uri = /vpc-routes
33052024-07-19T17:51:01.939ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34411
33062024-07-19T17:51:01.940ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 121 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:34411 req_id = 72ca5eb3-3992-47c5-8b1a-9034fc7b66ed response_code = 200 uri = /v2p
33072024-07-19T17:51:01.944ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37324
33082024-07-19T17:51:01.944ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 67 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:37324 req_id = da8bd239-4365-401d-be8b-0af629143dcc response_code = 200 uri = /v2p
33092024-07-19T17:51:02.019ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:44816
33102024-07-19T17:51:02.065ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 45395 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:44816 req_id = 7904554f-4321-43e2-bdc9-62f9a7205cf2 response_code = 204 uri = /switch-ports
33112024-07-19T17:51:02.066ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 661 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:44816 req_id = b2637611-685c-4e00-9119-cfc646abd73b response_code = 200 uri = /network-bootstore-config
33122024-07-19T17:51:02.102ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:49303
33132024-07-19T17:51:02.144ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 41939 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:49303 req_id = b5ce7420-de0f-4c45-8b2c-11e6c674c437 response_code = 204 uri = /switch-ports
33142024-07-19T17:51:02.145ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 542 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:49303 req_id = 75742359-7ff2-4db8-901c-1e29fd287db7 response_code = 200 uri = /network-bootstore-config
33152024-07-19T17:51:02.648ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:46883
33162024-07-19T17:51:02.649ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 965 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:46883 req_id = 9022e016-098d-4ad6-91a3-5e803975f55d response_code = 200 uri = /v2p
33172024-07-19T17:51:02.775ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:34215
33182024-07-19T17:51:02.775ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 72 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:34215 req_id = b8f0491d-9159-4932-a654-7a4b8c7feb61 response_code = 200 uri = /vpc-routes
33192024-07-19T17:51:02.788ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 138 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:34215 req_id = 237df109-37be-4e02-b0a3-c8a6b405458c response_code = 204 uri = /vpc-routes
33202024-07-19T17:51:02.955ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:33145
33212024-07-19T17:51:03.001ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 46121 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:33145 req_id = b860776f-4037-4ccd-9b1c-e6df8296e5ee response_code = 204 uri = /switch-ports
33222024-07-19T17:51:03.002ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 732 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:33145 req_id = bdbafc40-c9a1-46ce-9a30-fba7ca0c3bff response_code = 200 uri = /network-bootstore-config
33232024-07-19T17:51:14.560ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:46393
33242024-07-19T17:51:14.561ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 83 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:46393 req_id = 6b303ecb-b0da-4c21-b9df-94c142cf98a4 response_code = 200 uri = /baseboard
33252024-07-19T17:51:20.142ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 7742 local_addr = [fd00:1122:3344:101::1]:56984 method = GET remote_addr = [fd00:1122:3344:101::d]:57512 req_id = 4a382bdc-2bab-498a-a576-f749b16bee75 response_code = 200 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716 uri = /0c1108fb-cc91-4535-adf7-846d50ef8716
33262024-07-19T17:51:31.888ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:59640
33272024-07-19T17:51:31.888ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 69 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:59640 req_id = d9f48c3f-4fff-4c98-8b1b-b324f7cfe692 response_code = 200 uri = /vpc-routes
33282024-07-19T17:51:31.896ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 123 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:59640 req_id = 0a75f4e4-638c-4d39-bcbb-a0ebc8fc6e08 response_code = 204 uri = /vpc-routes
33292024-07-19T17:51:31.928ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:42830
33302024-07-19T17:51:31.928ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 121 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:42830 req_id = c39d62ab-e5de-4fbe-a9b3-43020b09e8c9 response_code = 200 uri = /v2p
33312024-07-19T17:51:31.987ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:51540
33322024-07-19T17:51:31.987ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 111 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:51540 req_id = 503e33d8-fa8d-4aa5-9b09-ce5c78253a44 response_code = 200 uri = /v2p
33332024-07-19T17:51:32.045ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:50690
33342024-07-19T17:51:32.045ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 67 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:50690 req_id = 534aec51-be5c-49df-8ca3-4766499e3565 response_code = 200 uri = /vpc-routes
33352024-07-19T17:51:32.053ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 112 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:50690 req_id = b60d3345-ff52-462e-91a1-4e895359a411 response_code = 204 uri = /vpc-routes
33362024-07-19T17:51:32.228ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:41171
33372024-07-19T17:51:32.279ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 50574 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:41171 req_id = 73b3b58d-3230-4dda-9e91-7d9bb2dcc438 response_code = 204 uri = /switch-ports
33382024-07-19T17:51:32.279ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:64078
33392024-07-19T17:51:32.327ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 47954 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:64078 req_id = 5d820f71-7508-4819-9a6f-bdcf9bd857b2 response_code = 204 uri = /switch-ports
33402024-07-19T17:51:32.327ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 135 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:41171 req_id = 2b80ec5c-5536-49a2-802d-25aaf5f276cd response_code = 200 uri = /network-bootstore-config
33412024-07-19T17:51:32.327ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 206 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:64078 req_id = a22c966c-039e-4e2c-81ee-43e11f40e746 response_code = 200 uri = /network-bootstore-config
33422024-07-19T17:51:32.826ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:62200
33432024-07-19T17:51:32.826ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 186 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:62200 req_id = 41cf05fb-f435-4263-9c2e-4391bbf13101 response_code = 200 uri = /v2p
33442024-07-19T17:51:32.917ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:54476
33452024-07-19T17:51:32.917ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 65 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:54476 req_id = dbe01a50-7210-4cac-aa88-8be45145774d response_code = 200 uri = /vpc-routes
33462024-07-19T17:51:32.925ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 718 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:54476 req_id = b2ead5d6-3fa3-481c-a71e-0d0e6cd75995 response_code = 204 uri = /vpc-routes
33472024-07-19T17:51:33.157ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:39458
33482024-07-19T17:51:33.204ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 46979 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:39458 req_id = cb3c89cc-6288-4ce4-a65f-f3f6ba9383f6 response_code = 204 uri = /switch-ports
33492024-07-19T17:51:33.204ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 101 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:39458 req_id = 64ba039c-9a95-4cbd-ae5c-48632cf6cb9d response_code = 200 uri = /network-bootstore-config
33502024-07-19T17:51:44.592ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:46497
33512024-07-19T17:51:44.592ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 100 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:46497 req_id = c7e406bb-e2f0-48d9-8ce9-d2fbcd9ee85c response_code = 200 uri = /baseboard
33522024-07-19T17:51:50.176ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 1618 local_addr = [fd00:1122:3344:101::1]:56984 method = GET remote_addr = [fd00:1122:3344:101::d]:57512 req_id = 19883cfd-aec1-4b77-8984-64b45b516051 response_code = 200 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716 uri = /0c1108fb-cc91-4535-adf7-846d50ef8716
33532024-07-19T17:52:01.864ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:51469
33542024-07-19T17:52:01.864ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 158 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:51469 req_id = e8b28e12-8b55-40a5-b500-05d69a62f1dc response_code = 200 uri = /v2p
33552024-07-19T17:52:01.866ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:49381
33562024-07-19T17:52:01.866ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 46 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:49381 req_id = 0323c7e8-d77c-4ea7-be8c-8db38a4cff0d response_code = 200 uri = /vpc-routes
33572024-07-19T17:52:01.874ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 713 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:49381 req_id = 92d50d4e-7c61-4900-a15b-e9062d177b09 response_code = 204 uri = /vpc-routes
33582024-07-19T17:52:01.964ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:57109
33592024-07-19T17:52:01.964ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 32 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:57109 req_id = 38907a50-8b2e-4219-9f9c-a58cc021c22a response_code = 200 uri = /vpc-routes
33602024-07-19T17:52:01.972ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 737 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:57109 req_id = 71e78500-2a03-4f5a-8c8f-1f62fa0e16f3 response_code = 204 uri = /vpc-routes
33612024-07-19T17:52:01.984ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36735
33622024-07-19T17:52:02.035ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 50581 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:36735 req_id = 6bd8c47a-8f02-4fa6-ba0c-ade3167727cf response_code = 204 uri = /switch-ports
33632024-07-19T17:52:02.035ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 680 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:36735 req_id = d6cdf3ac-aa46-454d-b275-21115ef8fee6 response_code = 200 uri = /network-bootstore-config
33642024-07-19T17:52:02.043ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:39743
33652024-07-19T17:52:02.044ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 178 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:39743 req_id = 729671a9-42b9-422b-bde8-089797ca2e0a response_code = 200 uri = /v2p
33662024-07-19T17:52:02.141ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:34396
33672024-07-19T17:52:02.187ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 45689 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:34396 req_id = befb97e3-f388-49a4-ab08-3c6e4eaf82a0 response_code = 204 uri = /switch-ports
33682024-07-19T17:52:02.187ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:34396 req_id = 49f919cd-6eb2-4847-9b40-e65f6b8d7bee response_code = 200 uri = /network-bootstore-config
33692024-07-19T17:52:02.586ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61071
33702024-07-19T17:52:02.587ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 855 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61071 req_id = 888ebaab-54c2-4359-86d5-57195db9001c response_code = 200 uri = /vpc-routes
33712024-07-19T17:52:02.596ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 149 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:61071 req_id = 22d42cc0-d62d-4294-a7a9-252e8d329e05 response_code = 204 uri = /vpc-routes
33722024-07-19T17:52:02.747ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61807
33732024-07-19T17:52:02.747ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 125 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61807 req_id = d4ba6264-4fe5-472f-b6b1-808c2f27b351 response_code = 200 uri = /v2p
33742024-07-19T17:52:02.853ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:57969
33752024-07-19T17:52:02.902ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 48449 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:57969 req_id = c52dab35-f83b-4878-bb03-f859933cc5e2 response_code = 204 uri = /switch-ports
33762024-07-19T17:52:02.902ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:57969 req_id = 29a46aea-18b7-4bde-add7-c8fcdf634119 response_code = 200 uri = /network-bootstore-config
33772024-07-19T17:52:14.626ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:42303
33782024-07-19T17:52:14.626ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 628 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:42303 req_id = 10fd0f6f-0fab-4b9d-b15a-a1dc7324fbb6 response_code = 200 uri = /baseboard
33792024-07-19T17:52:20.175ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 963 local_addr = [fd00:1122:3344:101::1]:56984 method = GET remote_addr = [fd00:1122:3344:101::d]:57512 req_id = abfc3581-7a57-4271-bf69-3c83d49fd81d response_code = 200 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716 uri = /0c1108fb-cc91-4535-adf7-846d50ef8716
33802024-07-19T17:52:32.038ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:60657
33812024-07-19T17:52:32.038ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 131 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:60657 req_id = 208a09d2-caff-4812-816b-32aea0a290d9 response_code = 200 uri = /vpc-routes
33822024-07-19T17:52:32.047ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 986 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:60657 req_id = 68d5a403-3726-4185-b32e-83ab01aa55c3 response_code = 204 uri = /vpc-routes
33832024-07-19T17:52:32.091ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:47276
33842024-07-19T17:52:32.091ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:673 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 005b2740-84d5-473e-ae59-aa79fd2fe5b6, kind: Service { id: 09fdc6fd-8297-436e-903d-b18710599203 }, name: Name("external-dns-09fdc6fd-8297-436e-903d-b18710599203"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 6])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 59731bd7-ffc9-493f-a33c-a1cf5ba2df30, kind: Service { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1 }, name: Name("external-dns-1eb90f9c-ba99-428e-885b-5fa2df7ae2f1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 32])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: c6edf68c-f558-4f51-9aed-6eb91fb3ad51, kind: Service { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52 }, name: Name("nexus-87d65a87-e1e5-43e4-9cba-b4afd5e9ac52"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 117])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e6898038-bf11-41c8-b971-6ff3eed6d38b, kind: Service { id: 6088410e-a8fc-4d33-b593-6385e863291f }, name: Name("nexus-6088410e-a8fc-4d33-b593-6385e863291f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 27])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: ffe6d41d-1349-42b2-a887-49d734eef422, kind: Service { id: 742e0330-77de-4723-a145-74751d41c73b }, name: Name("nexus-742e0330-77de-4723-a145-74751d41c73b"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 199, 184])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
33852024-07-19T17:52:32.091ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
33862024-07-19T17:52:32.091ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
33872024-07-19T17:52:32.091ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte0" rules = []
33882024-07-19T17:52:32.091ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
33892024-07-19T17:52:32.091ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
33902024-07-19T17:52:32.091ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
33912024-07-19T17:52:32.091ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 301 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:47276 req_id = 40a9f429-9ee8-4a9f-9a68-02eb09970e57 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
33922024-07-19T17:52:32.098ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61272
33932024-07-19T17:52:32.098ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 128 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:61272 req_id = f4df8d7d-4aaa-4951-b36b-cf88e65bd6ac response_code = 200 uri = /v2p
33942024-07-19T17:52:32.103ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:56119
33952024-07-19T17:52:32.104ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 1145 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:56119 req_id = e5831014-9b6f-428e-9b3b-ac7f20d0abe1 response_code = 200 uri = /v2p
33962024-07-19T17:52:32.135ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57277
33972024-07-19T17:52:32.136ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:673 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 005b2740-84d5-473e-ae59-aa79fd2fe5b6, kind: Service { id: 09fdc6fd-8297-436e-903d-b18710599203 }, name: Name("external-dns-09fdc6fd-8297-436e-903d-b18710599203"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 6])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 59731bd7-ffc9-493f-a33c-a1cf5ba2df30, kind: Service { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1 }, name: Name("external-dns-1eb90f9c-ba99-428e-885b-5fa2df7ae2f1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 32])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: c6edf68c-f558-4f51-9aed-6eb91fb3ad51, kind: Service { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52 }, name: Name("nexus-87d65a87-e1e5-43e4-9cba-b4afd5e9ac52"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 117])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e6898038-bf11-41c8-b971-6ff3eed6d38b, kind: Service { id: 6088410e-a8fc-4d33-b593-6385e863291f }, name: Name("nexus-6088410e-a8fc-4d33-b593-6385e863291f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 27])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: ffe6d41d-1349-42b2-a887-49d734eef422, kind: Service { id: 742e0330-77de-4723-a145-74751d41c73b }, name: Name("nexus-742e0330-77de-4723-a145-74751d41c73b"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 199, 184])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
33982024-07-19T17:52:32.136ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
33992024-07-19T17:52:32.136ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
34002024-07-19T17:52:32.136ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte0" rules = []
34012024-07-19T17:52:32.136ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34022024-07-19T17:52:32.136ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34032024-07-19T17:52:32.136ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34042024-07-19T17:52:32.136ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 960 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:57277 req_id = 3e167fdb-1ae8-43d8-aa3c-e01a4ed7df43 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
34052024-07-19T17:52:32.186ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34719
34062024-07-19T17:52:32.186ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 42 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:34719 req_id = 864e75b2-8eb4-48a9-abe9-83faa06e9cee response_code = 200 uri = /vpc-routes
34072024-07-19T17:52:32.194ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 84 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:34719 req_id = 5a95188a-bc16-40ff-aec7-5cf1723aa7a9 response_code = 204 uri = /vpc-routes
34082024-07-19T17:52:32.344ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33115
34092024-07-19T17:52:32.395ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 50649 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:33115 req_id = d7f2ec1b-f504-43c9-9977-9c38a8fe5a50 response_code = 204 uri = /switch-ports
34102024-07-19T17:52:32.395ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:43770
34112024-07-19T17:52:32.399ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 3756 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:33115 req_id = ba8defe1-9c2d-42f8-9ff7-ab278e449bc0 response_code = 200 uri = /network-bootstore-config
34122024-07-19T17:52:32.441ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 45775 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:43770 req_id = 453ec7a2-20d6-48b4-819f-e4ff1e5af78b response_code = 204 uri = /switch-ports
34132024-07-19T17:52:32.441ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 285 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:43770 req_id = 5b53ac9d-7689-45bf-a8c6-cdc1d3139f2b response_code = 200 uri = /network-bootstore-config
34142024-07-19T17:52:32.757ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:50858
34152024-07-19T17:52:32.757ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 180 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:50858 req_id = 192b9f44-4845-44d1-93ad-117d590fd462 response_code = 200 uri = /v2p
34162024-07-19T17:52:32.879ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:47177
34172024-07-19T17:52:32.880ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 928 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:47177 req_id = b01e5b82-dfab-49ad-b608-f8e917129b03 response_code = 200 uri = /vpc-routes
34182024-07-19T17:52:32.887ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 130 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:47177 req_id = b625311c-4e95-4534-9e9f-57fe02112a75 response_code = 204 uri = /vpc-routes
34192024-07-19T17:52:32.954ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:54576
34202024-07-19T17:52:32.954ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:673 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 005b2740-84d5-473e-ae59-aa79fd2fe5b6, kind: Service { id: 09fdc6fd-8297-436e-903d-b18710599203 }, name: Name("external-dns-09fdc6fd-8297-436e-903d-b18710599203"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 6])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 59731bd7-ffc9-493f-a33c-a1cf5ba2df30, kind: Service { id: 1eb90f9c-ba99-428e-885b-5fa2df7ae2f1 }, name: Name("external-dns-1eb90f9c-ba99-428e-885b-5fa2df7ae2f1"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 175, 32])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: c6edf68c-f558-4f51-9aed-6eb91fb3ad51, kind: Service { id: 87d65a87-e1e5-43e4-9cba-b4afd5e9ac52 }, name: Name("nexus-87d65a87-e1e5-43e4-9cba-b4afd5e9ac52"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 117])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e6898038-bf11-41c8-b971-6ff3eed6d38b, kind: Service { id: 6088410e-a8fc-4d33-b593-6385e863291f }, name: Name("nexus-6088410e-a8fc-4d33-b593-6385e863291f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 27])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: ffe6d41d-1349-42b2-a887-49d734eef422, kind: Service { id: 742e0330-77de-4723-a145-74751d41c73b }, name: Name("nexus-742e0330-77de-4723-a145-74751d41c73b"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 199, 184])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
34212024-07-19T17:52:32.954ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
34222024-07-19T17:52:32.954ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
34232024-07-19T17:52:32.954ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte0" rules = []
34242024-07-19T17:52:32.954ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34252024-07-19T17:52:32.954ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34262024-07-19T17:52:32.954ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34272024-07-19T17:52:32.954ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 329 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:54576 req_id = 8d130d79-1d28-4339-9bf5-7335b9a00686 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
34282024-07-19T17:52:33.225ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:62765
34292024-07-19T17:52:33.271ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 46679 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:62765 req_id = 91fd31e5-4c57-417f-a948-d95a37ce08e2 response_code = 204 uri = /switch-ports
34302024-07-19T17:52:33.272ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 772 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:62765 req_id = e5440c54-f664-458f-89e2-7fa76f4426e4 response_code = 200 uri = /network-bootstore-config
34312024-07-19T17:52:44.658ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:48523
34322024-07-19T17:52:44.659ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 95 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:48523 req_id = 1e860fb5-a4d7-4d77-8c83-358b7f93548d response_code = 200 uri = /baseboard
34332024-07-19T17:52:50.175ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 915 local_addr = [fd00:1122:3344:101::1]:56984 method = GET remote_addr = [fd00:1122:3344:101::d]:57512 req_id = cc426885-3f2d-44b1-b64a-bd21f141385a response_code = 200 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716 uri = /0c1108fb-cc91-4535-adf7-846d50ef8716
34342024-07-19T17:53:01.885ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:56204
34352024-07-19T17:53:01.886ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 168 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:56204 req_id = f53a60ae-e3a2-465e-983e-27ab07ba9708 response_code = 200 uri = /v2p
34362024-07-19T17:53:01.930ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:53645
34372024-07-19T17:53:01.930ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 55 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:53645 req_id = 131fabda-0b93-4975-ad34-6d3b98a7313a response_code = 200 uri = /vpc-routes
34382024-07-19T17:53:01.937ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 105 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:53645 req_id = 0808b8b2-6ddc-4c9f-b887-ff699be5b2f4 response_code = 204 uri = /vpc-routes
34392024-07-19T17:53:01.939ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:53092
34402024-07-19T17:53:01.939ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 82 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:53092 req_id = 0f2aef79-569f-44b2-86bb-16136d406cbe response_code = 200 uri = /v2p
34412024-07-19T17:53:01.968ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:55611
34422024-07-19T17:53:01.969ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 840 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:55611 req_id = b4835a90-ea9a-49ea-9bcd-02447c718f72 response_code = 200 uri = /vpc-routes
34432024-07-19T17:53:01.976ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 65 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:55611 req_id = 0b5317a0-8b71-4da1-a449-5c9b16506811 response_code = 204 uri = /vpc-routes
34442024-07-19T17:53:02.012ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:44199
34452024-07-19T17:53:02.061ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 49380 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:44199 req_id = 157f14e7-5e77-4973-8eea-30b6e8340aa2 response_code = 204 uri = /switch-ports
34462024-07-19T17:53:02.061ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 121 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:44199 req_id = 110f9d37-b23e-4ddd-a3ec-8dc12a800ad8 response_code = 200 uri = /network-bootstore-config
34472024-07-19T17:53:02.072ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:43628
34482024-07-19T17:53:02.119ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 47101 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:43628 req_id = a1305584-139e-4b6c-a26b-14c90291619b response_code = 204 uri = /switch-ports
34492024-07-19T17:53:02.120ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 172 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:43628 req_id = 4d681783-8830-461e-95a7-4f1c0ff25837 response_code = 200 uri = /network-bootstore-config
34502024-07-19T17:53:02.620ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:38240
34512024-07-19T17:53:02.620ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 232 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:38240 req_id = cee052a6-0074-4fdb-9175-67116f15003c response_code = 200 uri = /v2p
34522024-07-19T17:53:02.694ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40934
34532024-07-19T17:53:02.694ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 84 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40934 req_id = 3908a8c3-6556-4163-ad28-dcafebf166df response_code = 200 uri = /vpc-routes
34542024-07-19T17:53:02.701ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 133 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:40934 req_id = 370cda16-e5c3-4996-a2f8-1bf17899ac30 response_code = 204 uri = /vpc-routes
34552024-07-19T17:53:02.796ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:46555
34562024-07-19T17:53:02.846ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 49144 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:46555 req_id = 55ef6062-26e6-4ce6-a4b2-4597e1be917e response_code = 204 uri = /switch-ports
34572024-07-19T17:53:02.846ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:46555 req_id = 6e281a29-a018-4274-81cd-6189ede92ec1 response_code = 200 uri = /network-bootstore-config
34582024-07-19T17:53:14.690ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:65241
34592024-07-19T17:53:14.691ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 56 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:65241 req_id = 0464c8c0-7e7c-4431-9b04-0e008dee7b42 response_code = 200 uri = /baseboard
34602024-07-19T17:53:20.175ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 606 local_addr = [fd00:1122:3344:101::1]:56984 method = GET remote_addr = [fd00:1122:3344:101::d]:57512 req_id = 4bc8361a-8d8b-4a0e-9581-dabefb0762d4 response_code = 200 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716 uri = /0c1108fb-cc91-4535-adf7-846d50ef8716
34612024-07-19T17:53:31.920ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:57140
34622024-07-19T17:53:31.920ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 110 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:57140 req_id = 07d201c5-ccce-44c2-804d-40b950364e49 response_code = 200 uri = /vpc-routes
34632024-07-19T17:53:31.928ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:57140 req_id = 25b78066-b09c-4e6b-80db-75a22cb918bc response_code = 204 uri = /vpc-routes
34642024-07-19T17:53:31.943ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33735
34652024-07-19T17:53:31.943ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 122 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:33735 req_id = d6db9058-d08e-4715-ba27-f11970257efa response_code = 200 uri = /v2p
34662024-07-19T17:53:32.066ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:50404
34672024-07-19T17:53:32.066ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 62 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:50404 req_id = b4c75f6a-e1e8-435a-8333-719026f32b43 response_code = 200 uri = /vpc-routes
34682024-07-19T17:53:32.074ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 93 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:50404 req_id = 6b6f20cb-5495-458a-96ed-63c5a031a7b4 response_code = 204 uri = /vpc-routes
34692024-07-19T17:53:32.123ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:65123
34702024-07-19T17:53:32.123ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 114 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:65123 req_id = d73bcfc3-4510-48b2-ac1f-5fe18789f410 response_code = 200 uri = /v2p
34712024-07-19T17:53:32.256ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:49138
34722024-07-19T17:53:32.304ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 48352 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:49138 req_id = acd0460a-8064-4a56-93fa-f722fbb3b3b5 response_code = 204 uri = /switch-ports
34732024-07-19T17:53:32.304ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 100 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:49138 req_id = 43942805-d24d-49d7-b4d0-43cfc1035b61 response_code = 200 uri = /network-bootstore-config
34742024-07-19T17:53:32.381ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:41795
34752024-07-19T17:53:32.428ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 46403 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:41795 req_id = 8acb2ce5-0961-41cb-95c4-c6741d1ce119 response_code = 204 uri = /switch-ports
34762024-07-19T17:53:32.428ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 681 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:41795 req_id = 553e4877-2121-4fce-aa33-7bf8cf02e9ae response_code = 200 uri = /network-bootstore-config
34772024-07-19T17:53:32.743ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:43265
34782024-07-19T17:53:32.743ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 241 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:43265 req_id = e970ceb1-e14e-4cb8-af27-194eedf18903 response_code = 200 uri = /v2p
34792024-07-19T17:53:32.805ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:47804
34802024-07-19T17:53:32.806ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 831 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:47804 req_id = dda3507d-a201-4ea9-a105-aeadf45a9592 response_code = 200 uri = /vpc-routes
34812024-07-19T17:53:32.813ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 112 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:47804 req_id = 4e5ae20a-e3a5-41ad-a366-723bb1e99a99 response_code = 204 uri = /vpc-routes
34822024-07-19T17:53:33.089ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58224
34832024-07-19T17:53:33.134ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 45038 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:58224 req_id = 1e094d24-1581-4553-867b-55d2ec1ba107 response_code = 204 uri = /switch-ports
34842024-07-19T17:53:33.135ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 98 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58224 req_id = 1c74bf3d-2527-4719-b0c6-95ec1150a9ff response_code = 200 uri = /network-bootstore-config
34852024-07-19T17:53:44.723ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:41748
34862024-07-19T17:53:44.724ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 88 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:41748 req_id = e934dcb6-b677-484d-9285-49bd5cb87165 response_code = 200 uri = /baseboard
34872024-07-19T17:53:50.175ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 590 local_addr = [fd00:1122:3344:101::1]:56984 method = GET remote_addr = [fd00:1122:3344:101::d]:57512 req_id = dcfad900-f250-4c2c-863c-c4f5ca050675 response_code = 200 sled_id = 0c1108fb-cc91-4535-adf7-846d50ef8716 uri = /0c1108fb-cc91-4535-adf7-846d50ef8716
34882024-07-19T17:53:51.263ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:45941
34892024-07-19T17:53:51.263ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 270 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:45941 req_id = 08a3b64b-1dde-44a9-b0ef-8371b66566ab response_code = 200 uri = /v2p
34902024-07-19T17:53:51.264ZINFOSledAgent (PortManager): Mapping virtual NIC to physical host file = illumos-utils/src/opte/port_manager.rs:780 mapping = VirtualNetworkInterfaceHost { virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 248, 48, 19])), physical_host_ip: fd00:1122:3344:101::1, vni: Vni(11145136) }
34912024-07-19T17:53:51.264ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 137 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:45941 req_id = dca6206a-84d5-4471-8715-9612e1f710bb response_code = 204 uri = /v2p
34922024-07-19T17:53:51.389ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:55735
34932024-07-19T17:53:51.408ZINFOSledAgent (InstanceManager): ensuring instance is registered file = sled-agent/src/instance_manager.rs:577 hardware = InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: c14dc971-a97e-42ef-9aa1-1e3ad1cec2d6, kind: Instance { id: ccff4298-547e-428a-b327-016613e4386e }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 248, 48, 19])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(11145136), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.1.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.1.176), floating_ips: [], firewall_rules: [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: c14dc971-a97e-42ef-9aa1-1e3ad1cec2d6, kind: Instance { id: ccff4298-547e-428a-b327-016613e4386e }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 248, 48, 19])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(11145136), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: c14dc971-a97e-42ef-9aa1-1e3ad1cec2d6, kind: Instance { id: ccff4298-547e-428a-b327-016613e4386e }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 248, 48, 19])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(11145136), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(11145136))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: c14dc971-a97e-42ef-9aa1-1e3ad1cec2d6, kind: Instance { id: ccff4298-547e-428a-b327-016613e4386e }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 248, 48, 19])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(11145136), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-2a5942cc1c6", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: f09b670c-44c1-4800-8f7b-23ddfd8beb2e, read_only_parent: Some(Volume { block_size: 512, id: 493cd121-797a-4616-b69a-81ed09b6179e, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 3dcbfd03-a6f5-476e-ba57-1bc968f1fb16, key: Some("ygWrIc4lwZIpVeVyvMB0gYlWiOsDO4JXA2srWgfKhI4="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::17]:19001", "[fd00:1122:3344:101::14]:19001", "[fd00:1122:3344:101::13]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: f09b670c-44c1-4800-8f7b-23ddfd8beb2e, key: Some("AwWPqJfHsTuoVrRgUuzUUppRfaElgQAMSTZup36zums="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::17]:19002", "[fd00:1122:3344:101::16]:19001", "[fd00:1122:3344:101::15]:19000"] } }] } }], cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAACHuY7zWPNYAAC5jvNYAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAh7mO81jzWAAAuY7zWAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiJjY2ZmNDI5OC01NDdlLTQyOGEtYjMyNy0wMTY2MTNlNDM4NmUiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJSXhxak52cmtDSlNsanNSQ0luK21FNm50TzA3eUpybzM3TXdzQ2VVWXB6OSJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA") } instance_id = ccff4298-547e-428a-b327-016613e4386e instance_runtime = InstanceRuntimeState { propolis_id: Some(df1fc3a9-614e-45dd-b799-a14257326be0 (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-07-19T17:53:51.021735627Z } metadata = InstanceMetadata { silo_id: feca8d2d-3fe6-4796-88fc-7959b0c126c0, project_id: 63fa7974-6cf8-470b-93fd-0e60179cdd83 } propolis_addr = [fd00:1122:3344:101::1:0]:12400 propolis_id = df1fc3a9-614e-45dd-b799-a14257326be0 vmm_runtime = VmmRuntimeState { state: Starting, gen: Generation(1), time_updated: 2024-07-19T17:53:50.962217Z }
34942024-07-19T17:53:51.408ZINFOSledAgent (InstanceManager): registering new instance file = sled-agent/src/instance_manager.rs:611 instance_id = ccff4298-547e-428a-b327-016613e4386e (instance)
34952024-07-19T17:53:51.408ZINFOSledAgent (InstanceManager): initializing new Instance file = sled-agent/src/instance.rs:951 instance_id = ccff4298-547e-428a-b327-016613e4386e propolis_id = df1fc3a9-614e-45dd-b799-a14257326be0 state = InstanceInitialState { hardware: InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: c14dc971-a97e-42ef-9aa1-1e3ad1cec2d6, kind: Instance { id: ccff4298-547e-428a-b327-016613e4386e }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 248, 48, 19])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(11145136), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.1.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.1.176), floating_ips: [], firewall_rules: [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: c14dc971-a97e-42ef-9aa1-1e3ad1cec2d6, kind: Instance { id: ccff4298-547e-428a-b327-016613e4386e }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 248, 48, 19])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(11145136), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: c14dc971-a97e-42ef-9aa1-1e3ad1cec2d6, kind: Instance { id: ccff4298-547e-428a-b327-016613e4386e }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 248, 48, 19])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(11145136), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(11145136))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: c14dc971-a97e-42ef-9aa1-1e3ad1cec2d6, kind: Instance { id: ccff4298-547e-428a-b327-016613e4386e }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 248, 48, 19])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(11145136), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-2a5942cc1c6", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: f09b670c-44c1-4800-8f7b-23ddfd8beb2e, read_only_parent: Some(Volume { block_size: 512, id: 493cd121-797a-4616-b69a-81ed09b6179e, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 3dcbfd03-a6f5-476e-ba57-1bc968f1fb16, key: Some("ygWrIc4lwZIpVeVyvMB0gYlWiOsDO4JXA2srWgfKhI4="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::17]:19001", "[fd00:1122:3344:101::14]:19001", "[fd00:1122:3344:101::13]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: f09b670c-44c1-4800-8f7b-23ddfd8beb2e, key: Some("AwWPqJfHsTuoVrRgUuzUUppRfaElgQAMSTZup36zums="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::17]:19002", "[fd00:1122:3344:101::16]:19001", "[fd00:1122:3344:101::15]:19000"] } }] } }], cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAACHuY7zWPNYAAC5jvNYAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAh7mO81jzWAAAuY7zWAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiJjY2ZmNDI5OC01NDdlLTQyOGEtYjMyNy0wMTY2MTNlNDM4NmUiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJSXhxak52cmtDSlNsanNSQ0luK21FNm50TzA3eUpybzM3TXdzQ2VVWXB6OSJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA") }, instance_runtime: InstanceRuntimeState { propolis_id: Some(df1fc3a9-614e-45dd-b799-a14257326be0 (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-07-19T17:53:51.021735627Z }, vmm_runtime: VmmRuntimeState { state: Starting, gen: Generation(1), time_updated: 2024-07-19T17:53:50.962217Z }, propolis_addr: [fd00:1122:3344:101::1:0]:12400 }
34962024-07-19T17:53:51.410ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 21182 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:55735 req_id = 89119799-e295-4f7d-81b4-55abcd814afc response_code = 200 uri = /instances/ccff4298-547e-428a-b327-016613e4386e
34972024-07-19T17:53:51.521ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57914
34982024-07-19T17:53:51.523ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 1577 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:57914 req_id = 4e4243e1-f550-4fbc-a7a7-097d37d76596 response_code = 200 uri = /instances/ccff4298-547e-428a-b327-016613e4386e/state
34992024-07-19T17:53:51.529ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:426 port = Port { inner: PortInner(PortData { name: "opte6", ip: 172.30.0.5, mac: MacAddr6([168, 64, 37, 248, 48, 19]), slot: 0, vni: Vni { inner: 11145136 }, subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.0.1 } }) }
35002024-07-19T17:53:51.529ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
35012024-07-19T17:53:51.567ZINFOSledAgent (InstanceManager): Configuring new Omicron zone: oxz_propolis-server_df1fc3a9-614e-45dd-b799-a14257326be0 file = illumos-utils/src/zone.rs:316 instance_id = ccff4298-547e-428a-b327-016613e4386e
35022024-07-19T17:53:51.591ZINFOSledAgent (InstanceManager): Installing Omicron zone: oxz_propolis-server_df1fc3a9-614e-45dd-b799-a14257326be0 file = illumos-utils/src/zone.rs:354 instance_id = ccff4298-547e-428a-b327-016613e4386e
35032024-07-19T17:53:52.901ZINFOSledAgent (InstanceManager): Profile for oxz_propolis-server_df1fc3a9-614e-45dd-b799-a14257326be0: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="system/illumos/propolis-server"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlInstance0"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::1:0"/> <propval type="astring" name="listen_port" value="12400"/> <propval type="astring" name="metric_addr" value="[fd00:1122:3344:101::a]:12221"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34 instance_id = ccff4298-547e-428a-b327-016613e4386e
35042024-07-19T17:53:52.902ZINFOSledAgent (InstanceManager): Zone booting file = illumos-utils/src/running_zone.rs:465 instance_id = ccff4298-547e-428a-b327-016613e4386e zone = oxz_propolis-server_df1fc3a9-614e-45dd-b799-a14257326be0
35052024-07-19T17:53:53.156ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_df1fc3a9-614e-45dd-b799-a14257326be0") failed: Property not found. retry in 40.739378ms file = illumos-utils/src/svc.rs:36 instance_id = ccff4298-547e-428a-b327-016613e4386e zone = oxz_propolis-server_df1fc3a9-614e-45dd-b799-a14257326be0
35062024-07-19T17:53:53.209ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_df1fc3a9-614e-45dd-b799-a14257326be0") failed: Property not found. retry in 113.967665ms file = illumos-utils/src/svc.rs:36 instance_id = ccff4298-547e-428a-b327-016613e4386e zone = oxz_propolis-server_df1fc3a9-614e-45dd-b799-a14257326be0
35072024-07-19T17:53:53.337ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_df1fc3a9-614e-45dd-b799-a14257326be0") failed: Property not found. retry in 257.074395ms file = illumos-utils/src/svc.rs:36 instance_id = ccff4298-547e-428a-b327-016613e4386e zone = oxz_propolis-server_df1fc3a9-614e-45dd-b799-a14257326be0
35082024-07-19T17:53:53.606ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_df1fc3a9-614e-45dd-b799-a14257326be0") failed: Property not found. retry in 407.8322ms file = illumos-utils/src/svc.rs:36 instance_id = ccff4298-547e-428a-b327-016613e4386e zone = oxz_propolis-server_df1fc3a9-614e-45dd-b799-a14257326be0
35092024-07-19T17:53:54.024ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_df1fc3a9-614e-45dd-b799-a14257326be0") failed: Property not found. retry in 1.002608194s file = illumos-utils/src/svc.rs:36 instance_id = ccff4298-547e-428a-b327-016613e4386e zone = oxz_propolis-server_df1fc3a9-614e-45dd-b799-a14257326be0
35102024-07-19T17:53:55.038ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_df1fc3a9-614e-45dd-b799-a14257326be0") failed: Property not found. retry in 1.009936496s file = illumos-utils/src/svc.rs:36 instance_id = ccff4298-547e-428a-b327-016613e4386e zone = oxz_propolis-server_df1fc3a9-614e-45dd-b799-a14257326be0
35112024-07-19T17:53:56.058ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_df1fc3a9-614e-45dd-b799-a14257326be0") failed: Property not found. retry in 1.422586579s file = illumos-utils/src/svc.rs:36 instance_id = ccff4298-547e-428a-b327-016613e4386e zone = oxz_propolis-server_df1fc3a9-614e-45dd-b799-a14257326be0
35122024-07-19T17:53:57.490ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_df1fc3a9-614e-45dd-b799-a14257326be0") failed: Property not found. retry in 1.127368572s file = illumos-utils/src/svc.rs:36 instance_id = ccff4298-547e-428a-b327-016613e4386e zone = oxz_propolis-server_df1fc3a9-614e-45dd-b799-a14257326be0
35132024-07-19T17:53:58.645ZINFOSledAgent (InstanceManager): Started propolis in zone: oxz_propolis-server_df1fc3a9-614e-45dd-b799-a14257326be0 file = sled-agent/src/instance.rs:1460 instance_id = ccff4298-547e-428a-b327-016613e4386e
35142024-07-19T17:53:58.653ZINFOSledAgent (InstanceManager): Propolis SMF service is online file = sled-agent/src/instance.rs:1469 instance_id = ccff4298-547e-428a-b327-016613e4386e
35152024-07-19T17:53:58.688ZINFOSledAgent (InstanceManager): Propolis HTTP server online file = sled-agent/src/instance.rs:1483 instance_id = ccff4298-547e-428a-b327-016613e4386e
35162024-07-19T17:53:58.693ZINFOSledAgent (InstanceManager): Sending ensure request to propolis: InstanceEnsureRequest { cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAACHuY7zWPNYAAC5jvNYAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAh7mO81jzWAAAuY7zWAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiJjY2ZmNDI5OC01NDdlLTQyOGEtYjMyNy0wMTY2MTNlNDM4NmUiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJSXhxak52cmtDSlNsanNSQ0luK21FNm50TzA3eUpybzM3TXdzQ2VVWXB6OSJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"), disks: [DiskRequest { device: "nvme", name: "disk-2a5942cc1c6", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: f09b670c-44c1-4800-8f7b-23ddfd8beb2e, read_only_parent: Some(Volume { block_size: 512, id: 493cd121-797a-4616-b69a-81ed09b6179e, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 3dcbfd03-a6f5-476e-ba57-1bc968f1fb16, key: Some("ygWrIc4lwZIpVeVyvMB0gYlWiOsDO4JXA2srWgfKhI4="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::17]:19001", "[fd00:1122:3344:101::14]:19001", "[fd00:1122:3344:101::13]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: f09b670c-44c1-4800-8f7b-23ddfd8beb2e, key: Some("AwWPqJfHsTuoVrRgUuzUUppRfaElgQAMSTZup36zums="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::17]:19002", "[fd00:1122:3344:101::16]:19001", "[fd00:1122:3344:101::15]:19000"] } }] } }], migrate: None, nics: [NetworkInterfaceRequest { name: "opte6", slot: Slot(0) }], properties: InstanceProperties { bootrom_id: 00000000-0000-0000-0000-000000000000, description: "Test description", id: ccff4298-547e-428a-b327-016613e4386e, image_id: 00000000-0000-0000-0000-000000000000, memory: 1024, metadata: InstanceMetadata { project_id: 63fa7974-6cf8-470b-93fd-0e60179cdd83, silo_id: feca8d2d-3fe6-4796-88fc-7959b0c126c0 }, name: "localshark", vcpus: 2 } } file = sled-agent/src/instance.rs:698 instance_id = ccff4298-547e-428a-b327-016613e4386e
35172024-07-19T17:53:58.866ZINFOSledAgent (InstanceManager): result of instance_ensure call is Ok(InstanceEnsureResponse { migrate: None }) file = sled-agent/src/instance.rs:700 instance_id = ccff4298-547e-428a-b327-016613e4386e
35182024-07-19T17:53:58.867ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Starting), migration_status: NoMigration, time: 2024-07-19T17:53:58.867830726Z } file = sled-agent/src/instance.rs:584 instance_id = ccff4298-547e-428a-b327-016613e4386e
35192024-07-19T17:53:58.867ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:606 instance_id = ccff4298-547e-428a-b327-016613e4386e new_instance_state = InstanceRuntimeState { propolis_id: Some(df1fc3a9-614e-45dd-b799-a14257326be0 (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-07-19T17:53:51.021735627Z } new_vmm_state = VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-07-19T17:53:58.867830726Z } propolis_id = df1fc3a9-614e-45dd-b799-a14257326be0
35202024-07-19T17:53:58.867ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:503 instance_id = ccff4298-547e-428a-b327-016613e4386e state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(df1fc3a9-614e-45dd-b799-a14257326be0 (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-07-19T17:53:51.021735627Z }, propolis_id: df1fc3a9-614e-45dd-b799-a14257326be0 (propolis), vmm_state: VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-07-19T17:53:58.867830726Z }, migration_state: None }
35212024-07-19T17:53:58.965ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Running), migration_status: NoMigration, time: 2024-07-19T17:53:58.965184639Z } file = sled-agent/src/instance.rs:584 instance_id = ccff4298-547e-428a-b327-016613e4386e
35222024-07-19T17:53:58.965ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:606 instance_id = ccff4298-547e-428a-b327-016613e4386e new_instance_state = InstanceRuntimeState { propolis_id: Some(df1fc3a9-614e-45dd-b799-a14257326be0 (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-07-19T17:53:51.021735627Z } new_vmm_state = VmmRuntimeState { state: Running, gen: Generation(3), time_updated: 2024-07-19T17:53:58.965184639Z } propolis_id = df1fc3a9-614e-45dd-b799-a14257326be0
35232024-07-19T17:53:58.965ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:503 instance_id = ccff4298-547e-428a-b327-016613e4386e state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(df1fc3a9-614e-45dd-b799-a14257326be0 (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-07-19T17:53:51.021735627Z }, propolis_id: df1fc3a9-614e-45dd-b799-a14257326be0 (propolis), vmm_state: VmmRuntimeState { state: Running, gen: Generation(3), time_updated: 2024-07-19T17:53:58.965184639Z }, migration_state: None }
35242024-07-19T17:53:59.001ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:62107
35252024-07-19T17:53:59.001ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 55 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:62107 req_id = 124151e2-3290-4afa-ad5a-def894c85e9a response_code = 200 uri = /vpc-routes
35262024-07-19T17:53:59.048ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 827 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:62107 req_id = 3d7b9138-9e54-46a9-a755-8d3640f2e361 response_code = 204 uri = /vpc-routes
35272024-07-19T17:53:59.085ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50310
35282024-07-19T17:53:59.085ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 47 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50310 req_id = 0f0ed64e-56b3-4824-b89e-b8593b731f99 response_code = 200 uri = /vpc-routes
35292024-07-19T17:53:59.097ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 69 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:50310 req_id = af32d44d-d878-4550-816a-1141663c7b90 response_code = 204 uri = /vpc-routes
35302024-07-19T17:54:01.864ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:53214
35312024-07-19T17:54:01.864ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 116 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:53214 req_id = 013ba19a-52a3-408a-af4f-b01adf595d3c response_code = 200 uri = /vpc-routes
35322024-07-19T17:54:01.879ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 132 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:53214 req_id = 4381620b-4046-4bc3-a337-9cedda3a339e response_code = 204 uri = /vpc-routes
35332024-07-19T17:54:01.884ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:65101
35342024-07-19T17:54:01.884ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 120 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:65101 req_id = c3939abd-5916-4a18-87c4-f1e24b8d1d4b response_code = 200 uri = /v2p
35352024-07-19T17:54:01.893ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37907
35362024-07-19T17:54:01.894ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 1189 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:37907 req_id = eb99d0c4-f75a-4430-8698-9464467f25af response_code = 200 uri = /instances/ccff4298-547e-428a-b327-016613e4386e/state
35372024-07-19T17:54:01.896ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47507
35382024-07-19T17:54:01.896ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 190 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47507 req_id = 9d62c797-ae6f-4b4e-b574-9d7108c99dd5 response_code = 200 uri = /v2p
35392024-07-19T17:54:02.090ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:43503
35402024-07-19T17:54:02.141ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 50667 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:43503 req_id = 2c4597cd-3224-48ee-9848-984203002953 response_code = 204 uri = /switch-ports
35412024-07-19T17:54:02.141ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:43503 req_id = 64138975-7e35-472a-9efd-1a0c40e1378d response_code = 200 uri = /network-bootstore-config
35422024-07-19T17:54:02.182ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:44697
35432024-07-19T17:54:02.182ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 175 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:44697 req_id = bfc10dab-7f39-4b3d-807a-b82cd39b2fcf response_code = 200 uri = /instances/ccff4298-547e-428a-b327-016613e4386e/state
35442024-07-19T17:54:02.186ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:49885
35452024-07-19T17:54:02.187ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 792 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:49885 req_id = f8ff2b36-466b-4058-a38c-3d766c42016e response_code = 200 uri = /vpc-routes
35462024-07-19T17:54:02.202ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 129 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:49885 req_id = 79a42162-e45f-4b3e-b9a4-7463261eaeb3 response_code = 204 uri = /vpc-routes
35472024-07-19T17:54:02.307ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42826
35482024-07-19T17:54:02.356ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 49640 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:42826 req_id = 88f8b306-b3ba-4f2a-b7dc-241418d097ff response_code = 204 uri = /switch-ports
35492024-07-19T17:54:02.357ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 120 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:42826 req_id = 17f6907f-3537-4a99-85d0-7d2ceb12ee01 response_code = 200 uri = /network-bootstore-config
35502024-07-19T17:54:02.831ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:59488
35512024-07-19T17:54:02.831ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 185 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:59488 req_id = fed2c98c-d2f5-477c-98b0-28ebb44890f7 response_code = 200 uri = /instances/ccff4298-547e-428a-b327-016613e4386e/state
35522024-07-19T17:54:02.848ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:47109
35532024-07-19T17:54:02.848ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 80 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:47109 req_id = a00f11ac-8bcf-476b-8c22-ee4ba04ce3dc response_code = 200 uri = /vpc-routes
35542024-07-19T17:54:02.850ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58492
35552024-07-19T17:54:02.850ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 144 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58492 req_id = 59d9dd35-c551-4930-b50a-edd8dc4539bc response_code = 200 uri = /v2p
35562024-07-19T17:54:02.861ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 740 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:47109 req_id = fb11cacf-251c-45f8-aea5-b8691fabd1f3 response_code = 204 uri = /vpc-routes
35572024-07-19T17:54:02.903ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:34841
35582024-07-19T17:54:02.952ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 48447 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:34841 req_id = b6b01abc-73d8-4fd3-9f3f-13ef7a6a4b4a response_code = 204 uri = /switch-ports
35592024-07-19T17:54:02.953ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 815 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:34841 req_id = 9b281940-17da-4ac0-99a6-3449634c3d8b response_code = 200 uri = /network-bootstore-config
35602024-07-19T17:54:09.102ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:39415
35612024-07-19T17:54:09.102ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 547 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:39415 req_id = 366ccba2-179e-4f9f-9dc8-6374b66a40ac response_code = 200 uri = /instances/ccff4298-547e-428a-b327-016613e4386e/state
35622024-07-19T17:54:09.102ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Stopping), migration_status: NoMigration, time: 2024-07-19T17:54:09.102834165Z } file = sled-agent/src/instance.rs:584 instance_id = ccff4298-547e-428a-b327-016613e4386e
35632024-07-19T17:54:09.102ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:606 instance_id = ccff4298-547e-428a-b327-016613e4386e new_instance_state = InstanceRuntimeState { propolis_id: Some(df1fc3a9-614e-45dd-b799-a14257326be0 (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-07-19T17:53:51.021735627Z } new_vmm_state = VmmRuntimeState { state: Stopping, gen: Generation(5), time_updated: 2024-07-19T17:54:09.102834165Z } propolis_id = df1fc3a9-614e-45dd-b799-a14257326be0
35642024-07-19T17:54:09.102ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:503 instance_id = ccff4298-547e-428a-b327-016613e4386e state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(df1fc3a9-614e-45dd-b799-a14257326be0 (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-07-19T17:53:51.021735627Z }, propolis_id: df1fc3a9-614e-45dd-b799-a14257326be0 (propolis), vmm_state: VmmRuntimeState { state: Stopping, gen: Generation(5), time_updated: 2024-07-19T17:54:09.102834165Z }, migration_state: None }
35652024-07-19T17:54:09.156ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Destroyed), migration_status: NoMigration, time: 2024-07-19T17:54:09.156760378Z } file = sled-agent/src/instance.rs:584 instance_id = ccff4298-547e-428a-b327-016613e4386e
35662024-07-19T17:54:09.156ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:606 instance_id = ccff4298-547e-428a-b327-016613e4386e new_instance_state = InstanceRuntimeState { propolis_id: None, dst_propolis_id: None, migration_id: None, gen: Generation(5), time_updated: 2024-07-19T17:54:09.156760378Z } new_vmm_state = VmmRuntimeState { state: Destroyed, gen: Generation(6), time_updated: 2024-07-19T17:54:09.156760378Z } propolis_id = df1fc3a9-614e-45dd-b799-a14257326be0
35672024-07-19T17:54:09.156ZINFOSledAgent (InstanceManager): terminating VMM that has exited file = sled-agent/src/instance.rs:621 instance_id = ccff4298-547e-428a-b327-016613e4386e
35682024-07-19T17:54:09.157ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
35692024-07-19T17:54:09.160ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:404
35702024-07-19T17:54:09.161ZINFOSledAgent (ZoneBundler): creating zone bundle context = ZoneBundleContext { storage_dirs: ["/pool/int/20dccfdd-e68a-468d-8629-7f7e0834aa1c/debug/bundle/zone", "/pool/int/301424fa-c483-4ce9-92fb-3c8ee10e6f9f/debug/bundle/zone"], cause: TerminatedInstance, extra_log_dirs: ["/pool/ext/988e775a-ddc4-455d-bc4e-f2e3c228cc74/crypt/debug", "/pool/ext/f3be927c-0009-45ce-8f85-0c94f9b67b8f/crypt/debug", "/pool/ext/ef4ff4d0-0a2e-4db9-b6d2-7a22e69d7765/crypt/debug", "/pool/ext/4c50992d-c53d-42b7-a46f-5340e3aee556/crypt/debug", "/pool/ext/7c02949e-12b1-4d82-9ac3-2efd807aef95/crypt/debug", "/pool/ext/765bc25e-72d0-4d30-8cfe-a58b7db5c2e8/crypt/debug", "/pool/ext/696d009b-6d0c-4082-b81b-231c6c85d568/crypt/debug", "/pool/ext/311acf6c-db4a-40cb-bd1f-e33d2bc8747f/crypt/debug", "/pool/ext/41faf794-cdf9-49a0-b529-4c18e335c28c/crypt/debug"] } file = sled-agent/src/zone_bundle.rs:445 zone_name = oxz_propolis-server_df1fc3a9-614e-45dd-b799-a14257326be0
35712024-07-19T17:54:09.195ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:35741
35722024-07-19T17:54:09.195ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 173 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:35741 req_id = 6aadd0ad-53cf-4510-ba43-e46435c4b808 response_code = 200 uri = /vpc-routes
35732024-07-19T17:54:09.209ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 161 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:35741 req_id = 343dd6d0-79e2-4c8b-931b-86c9b018403b response_code = 204 uri = /vpc-routes
35742024-07-19T17:54:09.806ZINFOSledAgent (ZoneBundler): finished zone bundle file = sled-agent/src/zone_bundle.rs:1246 metadata = ZoneBundleMetadata { id: ZoneBundleId { zone_name: "oxz_propolis-server_df1fc3a9-614e-45dd-b799-a14257326be0", bundle_id: 6303eb3d-c4fd-4b0f-9f33-bede4fa5ca59 }, time_created: 2024-07-19T17:54:09.161753248Z, version: 0, cause: TerminatedInstance }
35752024-07-19T17:54:09.806ZWARNSledAgent (InstanceManager): Halting and removing zone: oxz_propolis-server_df1fc3a9-614e-45dd-b799-a14257326be0 file = sled-agent/src/instance.rs:793 instance_id = ccff4298-547e-428a-b327-016613e4386e
35762024-07-19T17:54:11.536ZINFOSledAgent (InstanceManager): halt_and_remove_logged: Previous zone state: Running file = illumos-utils/src/zone.rs:270 instance_id = ccff4298-547e-428a-b327-016613e4386e
35772024-07-19T17:54:11.552ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:906 id = RouterId { vni: Vni(11145136), kind: System }
35782024-07-19T17:54:11.552ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:906 id = RouterId { vni: Vni(11145136), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }
35792024-07-19T17:54:11.552ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:503 instance_id = ccff4298-547e-428a-b327-016613e4386e state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: None, dst_propolis_id: None, migration_id: None, gen: Generation(5), time_updated: 2024-07-19T17:54:09.156760378Z }, propolis_id: df1fc3a9-614e-45dd-b799-a14257326be0 (propolis), vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-07-19T17:54:11.552693243Z }, migration_state: None }
35802024-07-19T17:54:11.654ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:52746
35812024-07-19T17:54:11.654ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 209 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:52746 req_id = fa6d3f81-e7c2-40a2-8e04-0cf74fe6aabe response_code = 200 uri = /v2p
35822024-07-19T17:54:11.655ZINFOSledAgent (PortManager): Mapping virtual NIC to physical host file = illumos-utils/src/opte/port_manager.rs:780 mapping = VirtualNetworkInterfaceHost { virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 248, 48, 19])), physical_host_ip: fd00:1122:3344:101::1, vni: Vni(11145136) }
35832024-07-19T17:54:11.655ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 591 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:52746 req_id = ef7cecfc-e4a4-4e6b-94fc-52b8c8e721cd response_code = 204 uri = /v2p
35842024-07-19T17:54:11.852ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:503 instance_id = ccff4298-547e-428a-b327-016613e4386e state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: None, dst_propolis_id: None, migration_id: None, gen: Generation(5), time_updated: 2024-07-19T17:54:09.156760378Z }, propolis_id: df1fc3a9-614e-45dd-b799-a14257326be0 (propolis), vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-07-19T17:54:11.552693243Z }, migration_state: None }
35852024-07-19T17:54:11.857ZINFOSledAgent (InstanceManager): State monitoring task complete file = sled-agent/src/instance.rs:736 instance_id = ccff4298-547e-428a-b327-016613e4386e
35862024-07-19T17:54:11.892ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:61229
35872024-07-19T17:54:11.892ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 46 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:61229 req_id = cca1d286-7ffe-4c70-95b2-b278f00b1ae4 response_code = 200 uri = /vpc-routes
35882024-07-19T17:54:11.899ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 94 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:61229 req_id = 45063872-6458-4f54-8fb8-d67778c5db47 response_code = 204 uri = /vpc-routes
35892024-07-19T17:54:11.951ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:64693
35902024-07-19T17:54:11.951ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 35 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:64693 req_id = 360b338e-de3a-4d93-9487-92a9b43b8125 response_code = 200 uri = /vpc-routes
35912024-07-19T17:54:11.957ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 49 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:64693 req_id = 7b3c2f91-2746-4abf-aed9-258d8597994c response_code = 204 uri = /vpc-routes
35922024-07-19T17:54:12.080ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:55162
35932024-07-19T17:54:12.081ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 1001 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:55162 req_id = 8132341b-bb4e-49ff-a3b7-582942909283 response_code = 200 uri = /v2p
35942024-07-19T17:54:12.081ZINFOSledAgent (PortManager): Clearing mapping of virtual NIC to physical host file = illumos-utils/src/opte/port_manager.rs:820 mapping = VirtualNetworkInterfaceHost { virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 248, 48, 19])), physical_host_ip: fd00:1122:3344:101::1, vni: Vni(11145136) }
35952024-07-19T17:54:12.082ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 64 local_addr = [fd00:1122:3344:101::1]:12345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:55162 req_id = c26b19f2-3ca9-4102-b6bf-9436804bc274 response_code = 204 uri = /v2p
35962024-07-19T17:54:12.896ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42032
35972024-07-19T17:54:12.896ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 86 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:42032 req_id = 9509d037-cc9f-43ef-9495-45d10bd45ba5 response_code = 200 uri = /vpc-routes
35982024-07-19T17:54:12.903ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 125 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:42032 req_id = d0cd31b1-1064-407a-9f07-2011c169e60b response_code = 204 uri = /vpc-routes
35992024-07-19T17:54:14.755ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:41172
36002024-07-19T17:54:14.756ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 95 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:41172 req_id = 71729e1f-3a52-4b71-8112-2ecbf6642b80 response_code = 200 uri = /baseboard