Max level shown:
1 [ Aug 23 21:51:57 Enabled. ]
2 [ Aug 23 21:51:57 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]
3 [ Aug 23 21:51:57 Method "start" exited with status 0. ]
4 [ Aug 23 21:51:57 Rereading configuration. ]
5 [ Aug 23 21:51:57 No 'refresh' method defined. Treating as :true. ]
6 note: configured to log to "/dev/stdout"
72024-08-23T21:51:57.591ZINFOSledAgent: Ensuring contract reaper thread file = illumos-utils/src/running_zone.rs:113
82024-08-23T21:51:57.591ZINFOSledAgent (BootstrapAgentStartup): finding links ["net0", "net1"] file = sled-agent/src/bootstrap/pre_server.rs:255
92024-08-23T21:51:57.646ZINFOSledAgent (BootstrapAgentStartup): Starting mg-ddm service addr-objs = [AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }] file = sled-agent/src/bootstrap/pre_server.rs:262
102024-08-23T21:51:57.646ZINFOSledAgent (BootstrapAgentStartup): Importing mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:42 path = /opt/oxide/mg-ddm/pkg/ddm/manifest.xml
112024-08-23T21:51:57.759ZINFOSledAgent (BootstrapAgentStartup): Setting mg-ddm interfaces file = sled-agent/src/bootstrap/maghemite.rs:50 interfaces = ("net0/ll" "net1/ll")
122024-08-23T21:51:57.768ZINFOSledAgent (BootstrapAgentStartup): Enabling mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:56
132024-08-23T21:51:57.776ZINFOSledAgent (BootstrapAgentStartup): Ensuring zfs key directory exists file = sled-agent/src/bootstrap/pre_server.rs:272 path = /var/run/oxide/
142024-08-23T21:51:58.379ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fdb0:18c0:4d0d:9fb2::, width: 64 }
152024-08-23T21:51:58.518ZINFOSledAgent: Starting KeyManager file = sled-agent/src/long_running_tasks.rs:126
162024-08-23T21:51:58.518ZINFOSledAgent: Starting StorageManager file = sled-agent/src/long_running_tasks.rs:138
172024-08-23T21:51:58.518ZINFOSledAgent: Starting StorageMonitor file = sled-agent/src/long_running_tasks.rs:151
182024-08-23T21:51:58.518ZINFOSledAgent: Starting HardwareManager file = sled-agent/src/long_running_tasks.rs:173 nongimlet_observed_disks = [] sled_mode = Scrimlet { asic: SoftNpuZone }
192024-08-23T21:51:58.518ZINFOSledAgent (DumpSetup-worker): DumpSetup poll loop started. file = sled-agent/src/dump_setup.rs:520
202024-08-23T21:51:58.518ZINFOSledAgent (HardwareManager): Creating HardwareManager file = sled-hardware/src/illumos/mod.rs:716
212024-08-23T21:51:58.521ZINFOSledAgent (HardwareManager): Generated i86pc baseboard Pc { identifier: "buskin", model: "i86pc" } file = sled-hardware/src/illumos/mod.rs:623
222024-08-23T21:51:58.521ZWARNSledAgent (HardwareManager): Device is not a Gimlet (i86pc), proceeding with null hardware view file = sled-hardware/src/illumos/mod.rs:765
232024-08-23T21:51:58.521ZINFOSledAgent: Starting HardwareMonitor file = sled-agent/src/long_running_tasks.rs:187
242024-08-23T21:51:58.521ZINFOSledAgent (HardwareManager): Monitoring for hardware updates file = sled-hardware/src/illumos/mod.rs:824
252024-08-23T21:51:58.521ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_0.vdev
262024-08-23T21:51:58.521ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = None
272024-08-23T21:51:58.521ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:406
282024-08-23T21:51:58.521ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
292024-08-23T21:51:58.521ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } path = /scratch/m2_0.vdev
302024-08-23T21:51:58.531ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_0.vdev file = sled-hardware/src/disk.rs:360
312024-08-23T21:51:58.531ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:371 id = 156808b0-3956-455e-a8e1-2217e0afca54 (zpool)
322024-08-23T21:51:58.632ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:307 zpool = ZpoolName { id: 156808b0-3956-455e-a8e1-2217e0afca54 (zpool), kind: Internal }
332024-08-23T21:51:58.840ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:406
342024-08-23T21:51:58.840ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_1.vdev
352024-08-23T21:51:58.840ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
362024-08-23T21:51:58.840ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
372024-08-23T21:51:58.840ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:406
382024-08-23T21:51:58.840ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
392024-08-23T21:51:58.840ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } path = /scratch/m2_1.vdev
402024-08-23T21:51:58.849ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_1.vdev file = sled-hardware/src/disk.rs:360
412024-08-23T21:51:58.849ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:371 id = fe7217f3-1112-4ad5-a4fc-b58965332aab (zpool)
422024-08-23T21:51:58.938ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:307 zpool = ZpoolName { id: fe7217f3-1112-4ad5-a4fc-b58965332aab (zpool), kind: Internal }
432024-08-23T21:51:59.143ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_0.vdev
442024-08-23T21:51:59.143ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 156808b0-3956-455e-a8e1-2217e0afca54 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: fe7217f3-1112-4ad5-a4fc-b58965332aab (zpool), kind: Internal } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
452024-08-23T21:51:59.144ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
462024-08-23T21:51:59.144ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:406
472024-08-23T21:51:59.144ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
482024-08-23T21:51:59.144ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
492024-08-23T21:51:59.144ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_1.vdev
502024-08-23T21:51:59.144ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 156808b0-3956-455e-a8e1-2217e0afca54 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: fe7217f3-1112-4ad5-a4fc-b58965332aab (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
512024-08-23T21:51:59.144ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
522024-08-23T21:51:59.144ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
532024-08-23T21:51:59.144ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:406
542024-08-23T21:51:59.144ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
552024-08-23T21:51:59.144ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_2.vdev
562024-08-23T21:51:59.144ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 156808b0-3956-455e-a8e1-2217e0afca54 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: fe7217f3-1112-4ad5-a4fc-b58965332aab (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
572024-08-23T21:51:59.144ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:406
582024-08-23T21:51:59.144ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
592024-08-23T21:51:59.144ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
602024-08-23T21:51:59.144ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
612024-08-23T21:51:59.144ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_3.vdev
622024-08-23T21:51:59.144ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:406
632024-08-23T21:51:59.144ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 156808b0-3956-455e-a8e1-2217e0afca54 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: fe7217f3-1112-4ad5-a4fc-b58965332aab (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
642024-08-23T21:51:59.144ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
652024-08-23T21:51:59.144ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
662024-08-23T21:51:59.144ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
672024-08-23T21:51:59.144ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_4.vdev
682024-08-23T21:51:59.144ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 156808b0-3956-455e-a8e1-2217e0afca54 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: fe7217f3-1112-4ad5-a4fc-b58965332aab (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
692024-08-23T21:51:59.144ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
702024-08-23T21:51:59.144ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
712024-08-23T21:51:59.144ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:406
722024-08-23T21:51:59.144ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
732024-08-23T21:51:59.144ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_5.vdev
742024-08-23T21:51:59.144ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 156808b0-3956-455e-a8e1-2217e0afca54 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: fe7217f3-1112-4ad5-a4fc-b58965332aab (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
752024-08-23T21:51:59.144ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
762024-08-23T21:51:59.144ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
772024-08-23T21:51:59.144ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:406
782024-08-23T21:51:59.144ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
792024-08-23T21:51:59.144ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_6.vdev
802024-08-23T21:51:59.144ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 156808b0-3956-455e-a8e1-2217e0afca54 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: fe7217f3-1112-4ad5-a4fc-b58965332aab (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
812024-08-23T21:51:59.144ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:406
822024-08-23T21:51:59.144ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
832024-08-23T21:51:59.144ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
842024-08-23T21:51:59.144ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
852024-08-23T21:51:59.144ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_7.vdev
862024-08-23T21:51:59.144ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:406
872024-08-23T21:51:59.144ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 156808b0-3956-455e-a8e1-2217e0afca54 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: fe7217f3-1112-4ad5-a4fc-b58965332aab (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
882024-08-23T21:51:59.144ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
892024-08-23T21:51:59.144ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
902024-08-23T21:51:59.144ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
912024-08-23T21:51:59.144ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_8.vdev
922024-08-23T21:51:59.144ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:406
932024-08-23T21:51:59.144ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
942024-08-23T21:51:59.144ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 156808b0-3956-455e-a8e1-2217e0afca54 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: fe7217f3-1112-4ad5-a4fc-b58965332aab (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
952024-08-23T21:51:59.144ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
962024-08-23T21:51:59.144ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
972024-08-23T21:51:59.144ZINFOSledAgent: Waiting for boot disk file = sled-agent/src/long_running_tasks.rs:97
982024-08-23T21:51:59.144ZINFOSledAgent: Found boot disk DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-agent/src/long_running_tasks.rs:99
992024-08-23T21:51:59.144ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 156808b0-3956-455e-a8e1-2217e0afca54 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: fe7217f3-1112-4ad5-a4fc-b58965332aab (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
1002024-08-23T21:51:59.144ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
1012024-08-23T21:51:59.144ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
1022024-08-23T21:51:59.144ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
1032024-08-23T21:51:59.144ZINFOSledAgent: Starting Bootstore file = sled-agent/src/long_running_tasks.rs:211
1042024-08-23T21:51:59.144ZINFOSledAgent (bootstore): No ledger in /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1052024-08-23T21:51:59.144ZINFOSledAgent (bootstore): No ledger in /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1062024-08-23T21:51:59.144ZINFOSledAgent (bootstore): No ledger found. Loading Fsm::State as Uninitialized file = bootstore/src/schemes/v0/storage.rs:83 peer_id = pc-buskin-i86pc
1072024-08-23T21:51:59.144ZINFOSledAgent (bootstore): No ledger in /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1082024-08-23T21:51:59.144ZINFOSledAgent (bootstore): No ledger in /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1092024-08-23T21:51:59.144ZINFOSledAgent (bootstore): No ledger found for network config file = bootstore/src/schemes/v0/storage.rs:146 peer_id = pc-buskin-i86pc
1102024-08-23T21:51:59.144ZINFOSledAgent: Starting Bootstore DDMD poller file = sled-agent/src/long_running_tasks.rs:216
1112024-08-23T21:51:59.144ZINFOSledAgent: Starting ZoneBundler related tasks file = sled-agent/src/long_running_tasks.rs:231
1122024-08-23T21:51:59.155ZINFOSledAgent (ServiceManager): Creating ServiceManager file = sled-agent/src/services.rs:731
1132024-08-23T21:51:59.155ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:186 next_cleanup = Instant { tv_sec: 2070, tv_nsec: 161959806 } time_to_next_cleanup = 599.999931799s
1142024-08-23T21:51:59.155ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3861
1152024-08-23T21:51:59.155ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
1162024-08-23T21:51:59.155ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/sled-agent-request.json file = common/src/ledger.rs:188
1172024-08-23T21:51:59.155ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/sled-agent-request.json file = common/src/ledger.rs:188
1182024-08-23T21:51:59.158ZINFOSledAgent (dropshot (BootstrapAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:205 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80
1192024-08-23T21:51:59.158ZINFOSledAgent (SledAgentSprocketsServer): Started listening file = sled-agent/src/bootstrap/sprockets_server.rs:44 local_addr = [fdb0:18c0:4d0d:9fb2::1]:12346
1202024-08-23T21:51:59.158ZINFOSledAgent (RSS): Injecting RSS configuration: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.1.164, last: 10.151.1.173, }, ), ], external_dns_ips: [ 10.151.1.164, 10.151.1.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.1.100, vlan_id: None, local_pref: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.1.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, } file = sled-agent/src/rack_setup/service.rs:1053
1212024-08-23T21:51:59.158ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:73
1222024-08-23T21:51:59.158ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
1232024-08-23T21:51:59.158ZINFOSledAgent (RSS): No ledger in /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1242024-08-23T21:51:59.158ZINFOSledAgent (RSS): No ledger in /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1252024-08-23T21:51:59.158ZINFOSledAgent (RSS): RSS configuration has not been fully applied yet file = sled-agent/src/rack_setup/service.rs:1113
1262024-08-23T21:51:59.158ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
1272024-08-23T21:51:59.158ZINFOSledAgent (RSS): No ledger in /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/rss-sled-plan.json file = common/src/ledger.rs:188
1282024-08-23T21:51:59.158ZINFOSledAgent (RSS): No ledger in /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/rss-sled-plan.json file = common/src/ledger.rs:188
1292024-08-23T21:51:59.158ZINFOSledAgent (RSS): Creating new allocation plan file = sled-agent/src/rack_setup/service.rs:1154
1302024-08-23T21:51:59.158ZINFOSledAgent (RSS): Serializing plan file = sled-agent/src/rack_setup/plan/sled.rs:155
1312024-08-23T21:51:59.158ZINFOSledAgent (RSS): Creating plan for the sled at fdb0:18c0:4d0d:9fb2::1 file = sled-agent/src/rack_setup/plan/sled.rs:128
1322024-08-23T21:51:59.158ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
1332024-08-23T21:51:59.158ZINFOSledAgent (RSS): Writing ledger to /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1342024-08-23T21:51:59.159ZINFOSledAgent (RSS): Writing ledger to /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1352024-08-23T21:51:59.159ZINFOSledAgent (RSS): Sled plan written to storage: Plan { rack_id: da7dff3e-1935-4d21-ab58-793c0302e3ea, sleds: { [fdb0:18c0:4d0d:9fb2::1]:12346: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 5050fe14-0845-40b2-872d-d70a5b2b56be, rack_id: da7dff3e-1935-4d21-ab58-793c0302e3ea, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64, }, }, }, }, }, config: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.1.164, last: 10.151.1.173, }, ), ], external_dns_ips: [ 10.151.1.164, 10.151.1.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.1.100, vlan_id: None, local_pref: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.1.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, }, } file = sled-agent/src/rack_setup/plan/sled.rs:175
1362024-08-23T21:51:59.159ZINFOSledAgent (RSS): Writing Rack Network Configuration to bootstore file = sled-agent/src/rack_setup/service.rs:1187
1372024-08-23T21:51:59.159ZINFOSledAgent (bootstore): Attempting to update network config with generation: 1, current_generation: 0 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-buskin-i86pc
1382024-08-23T21:51:59.159ZINFOSledAgent (bootstore): Writing ledger to /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
1392024-08-23T21:51:59.159ZINFOSledAgent (bootstore): Writing ledger to /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
1402024-08-23T21:51:59.159ZINFOSledAgent (bootstore): Broadcasting network config with generation 1 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-buskin-i86pc
1412024-08-23T21:51:59.159ZINFOSledAgent (BootstrapAgentRssHandler): Received initialization request from RSS file = sled-agent/src/bootstrap/rss_handle.rs:217 request = StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 5050fe14-0845-40b2-872d-d70a5b2b56be, rack_id: da7dff3e-1935-4d21-ab58-793c0302e3ea, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } target_sled = [fdb0:18c0:4d0d:9fb2::1]:12346
1422024-08-23T21:51:59.159ZINFOSledAgent (SledAgentSprocketsServer): Accepted connection file = sled-agent/src/bootstrap/sprockets_server.rs:66 remote_addr = [fdb0:18c0:4d0d:9fb2::1]:58752
1432024-08-23T21:51:59.160ZINFOSledAgent (SledAgentMain): Loading Sled Agent: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 5050fe14-0845-40b2-872d-d70a5b2b56be, rack_id: da7dff3e-1935-4d21-ab58-793c0302e3ea, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } file = sled-agent/src/bootstrap/server.rs:359
1442024-08-23T21:51:59.160ZINFOSledAgent (SledAgentMain): KeyManager: using hardcoded secret retriever file = sled-agent/src/bootstrap/server.rs:374
1452024-08-23T21:51:59.160ZINFOSledAgent: setting up sled agent server file = sled-agent/src/server.rs:44
1462024-08-23T21:51:59.160ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:101::, width: 64 }
1472024-08-23T21:51:59.160ZINFOSledAgent (StorageManager): Received KeyManagerReady file = sled-storage/src/manager.rs:406
1482024-08-23T21:51:59.160ZINFOSledAgent (StorageManager): Loading storage config file = sled-storage/src/manager.rs:574
1492024-08-23T21:51:59.160ZINFOSledAgent (StorageManager): No ledger in /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_ledger
1502024-08-23T21:51:59.160ZINFOSledAgent (StorageManager): No ledger in /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_ledger
1512024-08-23T21:51:59.160ZINFOSledAgent (StorageManager): No ledger of physical disks exists file = sled-storage/src/manager.rs:563
1522024-08-23T21:51:59.160ZINFOSledAgent (StorageManager): KeyManager ready, but no ledger detected file = sled-storage/src/manager.rs:591
1532024-08-23T21:51:59.160ZINFOSledAgent (StorageManager): automatically managing disks file = sled-storage/src/manager.rs:395
1542024-08-23T21:51:59.160ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:73
1552024-08-23T21:51:59.160ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
1562024-08-23T21:51:59.160ZINFOSledAgent (StorageManager): Successfully synchronized disks without error file = sled-storage/src/manager.rs:481 result = DisksManagementResult { status: [] }
1572024-08-23T21:51:59.162ZINFOSledAgent (ServiceManager): Enabling switch zone (new) file = sled-agent/src/services.rs:4158
1582024-08-23T21:51:59.162ZINFOSledAgent (ServiceManager): Starting switch zone file = sled-agent/src/services.rs:4540
1592024-08-23T21:51:59.192ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
1602024-08-23T21:51:59.211ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:316
1612024-08-23T21:51:59.229ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:354
1622024-08-23T21:51:59.242ZINFOSledAgent: SledAgent::new(..) starting file = sled-agent/src/sled_agent.rs:388 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
1632024-08-23T21:51:59.242ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
1642024-08-23T21:51:59.242ZINFOSledAgent: Requested swap device of size 64 GiB file = sled-agent/src/sled_agent.rs:400 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
1652024-08-23T21:51:59.242ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1662024-08-23T21:51:59.245ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 138.22301ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "62604daf-1d0a-47b6-9cc2-7f7141a57498", "content-length": "124", "date": "Fri, 23 Aug 2024 21:51:58 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "62604daf-1d0a-47b6-9cc2-7f7141a57498" } file = clients/ddm-admin-client/src/lib.rs:177
1672024-08-23T21:51:59.256ZINFOSledAgent: attempting to create encrypted zvol: name="oxi_156808b0-3956-455e-a8e1-2217e0afca54/swap", size_gb=64 file = sled-agent/src/swap_device.rs:153
1682024-08-23T21:51:59.280ZINFOSledAgent: successfully created encrypted zvol: name="oxi_156808b0-3956-455e-a8e1-2217e0afca54/swap", size_gb=64 file = sled-agent/src/swap_device.rs:249
1692024-08-23T21:51:59.280ZINFOSledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_156808b0-3956-455e-a8e1-2217e0afca54/swap" file = sled-agent/src/swap_device.rs:96
1702024-08-23T21:51:59.285ZINFOSledAgent: Mounting backing filesystems file = sled-agent/src/sled_agent.rs:415 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
1712024-08-23T21:51:59.285ZINFOSledAgent (BackingFs): Processing fmd file = sled-agent/src/backing_fs.rs:126
1722024-08-23T21:51:59.285ZINFOSledAgent (BackingFs): Ensuring dataset oxi_156808b0-3956-455e-a8e1-2217e0afca54/backing/fmd file = sled-agent/src/backing_fs.rs:136
1732024-08-23T21:51:59.362ZINFOSledAgent (BackingFs): Stopping service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:170
1742024-08-23T21:51:59.384ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1752024-08-23T21:51:59.385ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 175.85711ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "71fd074a-c6e3-4ee0-9440-5532952e8966", "content-length": "124", "date": "Fri, 23 Aug 2024 21:51:59 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "71fd074a-c6e3-4ee0-9440-5532952e8966" } file = clients/ddm-admin-client/src/lib.rs:177
1762024-08-23T21:51:59.562ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1772024-08-23T21:51:59.562ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 198.267209ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "085723c6-fe6a-4818-80a7-585ff16afcd3", "content-length": "124", "date": "Fri, 23 Aug 2024 21:51:59 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "085723c6-fe6a-4818-80a7-585ff16afcd3" } file = clients/ddm-admin-client/src/lib.rs:177
1782024-08-23T21:51:59.582ZINFOSledAgent (BackingFs): Mounting oxi_156808b0-3956-455e-a8e1-2217e0afca54/backing/fmd on /var/fm/fmd file = sled-agent/src/backing_fs.rs:178
1792024-08-23T21:51:59.596ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/rsrc file = sled-agent/src/backing_fs.rs:186
1802024-08-23T21:51:59.596ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/ckpt file = sled-agent/src/backing_fs.rs:186
1812024-08-23T21:51:59.596ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/xprt file = sled-agent/src/backing_fs.rs:186
1822024-08-23T21:51:59.596ZINFOSledAgent (BackingFs): Starting service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:192
1832024-08-23T21:51:59.762ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1842024-08-23T21:51:59.763ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 239.275009ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "bac20cc6-bd5f-44d7-af52-2539d82ceb61", "content-length": "124", "date": "Fri, 23 Aug 2024 21:51:59 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "bac20cc6-bd5f-44d7-af52-2539d82ceb61" } file = clients/ddm-admin-client/src/lib.rs:177
1852024-08-23T21:52:00.002ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1862024-08-23T21:52:00.003ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 185.204786ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "f585cf53-5379-4f25-b2ef-72adc4c4870e", "content-length": "124", "date": "Fri, 23 Aug 2024 21:51:59 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "f585cf53-5379-4f25-b2ef-72adc4c4870e" } file = clients/ddm-admin-client/src/lib.rs:177
1872024-08-23T21:52:00.190ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1882024-08-23T21:52:00.191ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 135.48865ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "da6f1b6e-4b9e-4149-973b-57f7a8b759ec", "content-length": "124", "date": "Fri, 23 Aug 2024 21:51:59 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "da6f1b6e-4b9e-4149-973b-57f7a8b759ec" } file = clients/ddm-admin-client/src/lib.rs:177
1892024-08-23T21:52:00.323ZINFOSledAgent: using '[AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }]' as data links for xde driver file = illumos-utils/src/opte/illumos.rs:90 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
1902024-08-23T21:52:00.328ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1912024-08-23T21:52:00.329ZINFOSledAgent (dropshot): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:205 local_addr = [fd00:1122:3344:101::1]:46625 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
1922024-08-23T21:52:00.329ZINFOSledAgent (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:73 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
1932024-08-23T21:52:00.329ZINFOSledAgent (producer-server): starting oximeter metric producer server address = [fd00:1122:3344:101::1]:46625 file = oximeter/producer/src/lib.rs:283 interval = 30s producer_id = 5050fe14-0845-40b2-872d-d70a5b2b56be sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
1942024-08-23T21:52:00.330ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 203.773123ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
1952024-08-23T21:52:00.331ZINFOSledAgent (VmmReservoirManager): 50% of 102952026112 physical ram = 51476013056 bytes) file = sled-agent/src/vmm_reservoir.rs:267 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
1962024-08-23T21:52:00.331ZINFOSledAgent (VmmReservoirManager): Setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:275 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
1972024-08-23T21:52:00.331ZINFOSledAgent (ServiceManager): sled agent started file = sled-agent/src/services.rs:915 underlay_address = fd00:1122:3344:101::1
1982024-08-23T21:52:00.331ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:73
1992024-08-23T21:52:00.536ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 263.523668ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
2002024-08-23T21:52:00.802ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 1.03732647s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
2012024-08-23T21:52:01.842ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 1.977895864s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
2022024-08-23T21:52:02.681ZINFOSledAgent (ServiceManager): Setting up dendrite service file = sled-agent/src/services.rs:2634
2032024-08-23T21:52:02.681ZINFOSledAgent (ServiceManager): Setting up lldpd service file = sled-agent/src/services.rs:2852
2042024-08-23T21:52:02.681ZINFOSledAgent (ServiceManager): Setting up MGS service file = sled-agent/src/services.rs:2504
2052024-08-23T21:52:02.681ZINFOSledAgent (ServiceManager): Setting up wicketd service file = sled-agent/src/services.rs:2551
2062024-08-23T21:52:02.681ZINFOSledAgent (ServiceManager): Setting up mgd service file = sled-agent/src/services.rs:2933
2072024-08-23T21:52:02.681ZINFOSledAgent (ServiceManager): Setting up mg-ddm service file = sled-agent/src/services.rs:2975
2082024-08-23T21:52:02.683ZINFOSledAgent (ServiceManager): Setting up tfport service file = sled-agent/src/services.rs:2777
2092024-08-23T21:52:02.684ZINFOSledAgent (ServiceManager): Setting up Simulated SP service file = sled-agent/src/services.rs:2545
2102024-08-23T21:52:02.684ZINFOSledAgent (ServiceManager): Profile for oxz_switch: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/mgs"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12225'/> <propval type="astring" name="id" value='81c753a5-2021-414f-8260-50eeee2f0957'/> <propval type="astring" name="rack_id" value='da7dff3e-1935-4d21-ab58-793c0302e3ea'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/wicketd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12226'/> <propval type="astring" name="artifact-address" value='[fdb0:18c0:4d0d:9fb2::2]:12227'/> <propval type="astring" name="baseboard-file" value='/opt/oxide/baseboard.json'/> <propval type="astring" name="mgs-address" value='[::1]:12225'/> <propval type="astring" name="nexus-proxy-address" value='[::]:12229'/> <propval type="astring" name="rack-subnet" value='fd00:1122:3344::'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/switch_zone_setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="gz_local_link_addr" value='fe80::8:20ff:feeb:15d1'/> <property type="astring" name="link_local_links"> <astring_list> <value_node value="tfportrear0_0" /> <value_node value="oxBootstrap0" /> </astring_list> </property> <propval type="astring" name="bootstrap_addr" value='fdb0:18c0:4d0d:9fb2::2'/> <propval type="astring" name="bootstrap_vnic" value='oxBootstrap0'/> <propval type="astring" name="baseboard_info" value='{"type":"pc","identifier":"buskin","model":"i86pc"}'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/dendrite"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_id" value='5050fe14-0845-40b2-872d-d70a5b2b56be'/> <propval type="astring" name="rack_id" value='da7dff3e-1935-4d21-ab58-793c0302e3ea'/> <propval type="astring" name="address" value='[::1]:12224'/> <propval type="astring" name="front_ports" value='1'/> <propval type="astring" name="rear_ports" value='1'/> <propval type="astring" name="port_config" value='/opt/oxide/dendrite/misc/softnpu_single_sled_config.toml'/> <propval type="astring" name="mgmt" value='uds'/> <propval type="astring" name="uds_path" value='/opt/softnpu/stuff'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/tfport"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="host" value='[::1]'/> <propval type="astring" name="port" value='12224'/> <propval type="astring" name="flags" value='--sync-only'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/lldpd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="board_rev" value='softnpu_front_1_rear_1'/> <propval type="astring" name="scrimlet_id" value='buskin'/> <propval type="astring" name="scrimlet_model" value='i86pc'/> <propval type="astring" name="address" value='[::1]:12230'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/pumpkind"> </service> <service version="1" type="service" name="oxide/mgd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_uuid" value='5050fe14-0845-40b2-872d-d70a5b2b56be'/> <propval type="astring" name="rack_uuid" value='da7dff3e-1935-4d21-ab58-793c0302e3ea'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/mg-ddm"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="mode" value='transit'/> <propval type="astring" name="dendrite" value='true'/> <propval type="astring" name="sled_uuid" value='5050fe14-0845-40b2-872d-d70a5b2b56be'/> <propval type="astring" name="rack_uuid" value='da7dff3e-1935-4d21-ab58-793c0302e3ea'/> <propval type="astring" name="interfaces" value='tfportrear0_0/ll'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/uplink"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
2112024-08-23T21:52:02.684ZINFOSledAgent (ServiceManager): Booting oxz_switch zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_switch
2122024-08-23T21:52:03.062ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 53.449223ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2132024-08-23T21:52:03.239ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 84.898955ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2142024-08-23T21:52:03.430ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 212.538565ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2152024-08-23T21:52:03.666ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 592.577042ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2162024-08-23T21:52:03.821ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 3.88271509s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
2172024-08-23T21:52:04.394ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.023242138s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2182024-08-23T21:52:05.488ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 703.928318ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2192024-08-23T21:52:06.381ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 682.468545ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2202024-08-23T21:52:07.280ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.243357093s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2212024-08-23T21:52:07.706ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 6.636376331s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
2222024-08-23T21:52:09.009ZINFOSledAgent (dropshot (SledAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:205 local_addr = [fd00:1122:3344:101::1]:12345
2232024-08-23T21:52:09.009ZINFOSledAgent (SledAgentMain): Sled Agent loaded; recording configuration file = sled-agent/src/bootstrap/server.rs:429
2242024-08-23T21:52:09.009ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
2252024-08-23T21:52:09.009ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2262024-08-23T21:52:09.009ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2272024-08-23T21:52:09.009ZINFOSledAgent (HardwareMonitor): Sled Agent Started file = sled-agent/src/hardware_monitor.rs:138
2282024-08-23T21:52:09.009ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = Some((fd00:1122:3344:101::2, Some(RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56 }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [PortConfigV2 { routes: [RouteConfig { destination: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), nexthop: 10.151.1.100, vlan_id: None, local_pref: None }], addresses: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None }], bgp: [], bfd: [] })))
2292024-08-23T21:52:09.009ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3861
2302024-08-23T21:52:09.009ZINFOSledAgent (SledAgentSprocketsServer): Connection closed file = sled-agent/src/bootstrap/sprockets_server.rs:73 remote_addr = [fdb0:18c0:4d0d:9fb2::1]:58752
2312024-08-23T21:52:09.009ZINFOSledAgent (BootstrapAgentRssHandler): Peer agent initialized file = sled-agent/src/bootstrap/rss_handle.rs:108 peer = [fdb0:18c0:4d0d:9fb2::1]:12346
2322024-08-23T21:52:09.009ZINFOSledAgent (BootstrapAgentRssHandler): Initialized sled agent file = sled-agent/src/bootstrap/rss_handle.rs:232 target_sled = [fdb0:18c0:4d0d:9fb2::1]:12346
2332024-08-23T21:52:09.242ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
2342024-08-23T21:52:09.242ZINFOSledAgent (RSS): No ledger in /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/rss-service-plan-v3.json file = common/src/ledger.rs:188
2352024-08-23T21:52:09.242ZINFOSledAgent (RSS): No ledger in /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/rss-service-plan-v3.json file = common/src/ledger.rs:188
2362024-08-23T21:52:09.242ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
2372024-08-23T21:52:09.242ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
2382024-08-23T21:52:09.247ZINFOSledAgent (ServiceManager): Re-enabling running switch zone (new address) file = sled-agent/src/services.rs:4182 new = [fd00:1122:3344:101::2, ::1] old = [::1]
2392024-08-23T21:52:09.247ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists file = sled-agent/src/services.rs:4198
2402024-08-23T21:52:09.247ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::2, prefix: 64 })) file = illumos-utils/src/running_zone.rs:534 zone = oxz_switch
2412024-08-23T21:52:09.500ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:46501
2422024-08-23T21:52:09.500ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
2432024-08-23T21:52:09.500ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 323 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:46501 req_id = 26071094-3b45-4b8d-833e-9e89ba5bd968 response_code = 200 uri = /inventory
2442024-08-23T21:52:09.531ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:37130
2452024-08-23T21:52:09.531ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 57 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:37130 req_id = a4728655-1bed-4078-9133-1612fadf1d57 response_code = 200 uri = /sled-role
2462024-08-23T21:52:09.531ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
2472024-08-23T21:52:09.531ZINFOSledAgent (RSS): Writing ledger to /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/.rss-service-plan-v3.json.tmp file = common/src/ledger.rs:199
2482024-08-23T21:52:09.532ZINFOSledAgent (RSS): Writing ledger to /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/.rss-service-plan-v3.json.tmp file = common/src/ledger.rs:199
2492024-08-23T21:52:09.532ZINFOSledAgent (RSS): Service plan written to storage file = sled-agent/src/rack_setup/plan/service.rs:884
2502024-08-23T21:52:09.565ZINFOSledAgent (RSS): attempting to set up sled's storage: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: 11713d4c-ac38-409f-b44b-50d6a207b503, pool_id: c4828395-89b1-4653-93a7-55b242871c48 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: cce62975-3bcc-483e-bd28-228e5a44011a, pool_id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: 940fedf0-b769-445c-983e-f4f5c9010dce, pool_id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: 4ee4b373-3198-4629-b431-9a7495b69ca1, pool_id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: ac5eacfe-7b2a-4ada-b65b-050c9807cd1e, pool_id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: dd1eb680-3131-4e02-869e-de5fafaf42f9, pool_id: dad70eae-b83f-4c3a-a32e-2a6a8946803c (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 7dd96546-1f9b-41d8-bb80-b12a0f1de1a5, pool_id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: 8e3c003d-8926-48ad-98e1-438bed02e673, pool_id: e1a016b1-16ad-48d4-a26f-2e916f51bc80 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: bc3aa74f-31f1-4cb7-a638-01dc0627d7f2, pool_id: 7ac03450-9a27-4ec4-9669-9eb798da4419 (zpool) }] } file = sled-agent/src/rack_setup/service.rs:357 sled_address = [fd00:1122:3344:101::1]:12345
2512024-08-23T21:52:09.565ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:59727
2522024-08-23T21:52:09.565ZINFOSledAgent: physical disks ensure file = sled-agent/src/sled_agent.rs:833 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
2532024-08-23T21:52:09.565ZINFOSledAgent (StorageManager): Received OmicronPhysicalDisksEnsure { config: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: 11713d4c-ac38-409f-b44b-50d6a207b503, pool_id: c4828395-89b1-4653-93a7-55b242871c48 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: cce62975-3bcc-483e-bd28-228e5a44011a, pool_id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: 940fedf0-b769-445c-983e-f4f5c9010dce, pool_id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: 4ee4b373-3198-4629-b431-9a7495b69ca1, pool_id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: ac5eacfe-7b2a-4ada-b65b-050c9807cd1e, pool_id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: dd1eb680-3131-4e02-869e-de5fafaf42f9, pool_id: dad70eae-b83f-4c3a-a32e-2a6a8946803c (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 7dd96546-1f9b-41d8-bb80-b12a0f1de1a5, pool_id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: 8e3c003d-8926-48ad-98e1-438bed02e673, pool_id: e1a016b1-16ad-48d4-a26f-2e916f51bc80 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: bc3aa74f-31f1-4cb7-a638-01dc0627d7f2, pool_id: 7ac03450-9a27-4ec4-9669-9eb798da4419 (zpool) }] }, tx: ... } file = sled-storage/src/manager.rs:406
2542024-08-23T21:52:09.565ZINFOSledAgent (StorageManager): No ledger in /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2552024-08-23T21:52:09.565ZINFOSledAgent (StorageManager): No ledger in /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2562024-08-23T21:52:09.565ZINFOSledAgent (StorageManager): No previously-stored 'requested disks', creating new ledger file = sled-storage/src/manager.rs:645 request = omicron_physical_disks_ensure
2572024-08-23T21:52:09.565ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
2582024-08-23T21:52:09.565ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:342
2592024-08-23T21:52:09.565ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2602024-08-23T21:52:09.565ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } path = /scratch/u2_0.vdev
2612024-08-23T21:52:09.742ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_0.vdev file = sled-hardware/src/disk.rs:360
2622024-08-23T21:52:09.742ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = c4828395-89b1-4653-93a7-55b242871c48 (zpool)
2632024-08-23T21:52:10.210ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/5050fe14-0845-40b2-872d-d70a5b2b56be): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
2642024-08-23T21:52:10.689ZINFOSledAgent (VmmReservoirManager): Finished setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:279 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
2652024-08-23T21:52:10.720ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:307 zpool = ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }
2662024-08-23T21:52:10.729ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:351
2672024-08-23T21:52:10.729ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:353
2682024-08-23T21:52:10.729ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:357
2692024-08-23T21:52:10.729ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:359
2702024-08-23T21:52:10.729ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2712024-08-23T21:52:10.729ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2722024-08-23T21:52:10.729ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:371
2732024-08-23T21:52:10.765ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2742024-08-23T21:52:10.773ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_c4828395-89b1-4653-93a7-55b242871c48/crypt/zone file = sled-storage/src/dataset.rs:414
2752024-08-23T21:52:10.895ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_c4828395-89b1-4653-93a7-55b242871c48 file = sled-storage/src/dataset.rs:499
2762024-08-23T21:52:10.903ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:440
2772024-08-23T21:52:10.903ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:357
2782024-08-23T21:52:10.903ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:342
2792024-08-23T21:52:10.903ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2802024-08-23T21:52:10.903ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } path = /scratch/u2_1.vdev
2812024-08-23T21:52:10.913ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_1.vdev file = sled-hardware/src/disk.rs:360
2822024-08-23T21:52:10.913ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool)
2832024-08-23T21:52:11.008ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:307 zpool = ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }
2842024-08-23T21:52:11.016ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:351
2852024-08-23T21:52:11.016ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:353
2862024-08-23T21:52:11.016ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:357
2872024-08-23T21:52:11.016ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:359
2882024-08-23T21:52:11.016ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2892024-08-23T21:52:11.017ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2902024-08-23T21:52:11.017ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:371
2912024-08-23T21:52:11.053ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2922024-08-23T21:52:11.061ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_c4fe4e01-40de-4c9e-857a-e6870bd84153/crypt/zone file = sled-storage/src/dataset.rs:414
2932024-08-23T21:52:11.181ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_c4fe4e01-40de-4c9e-857a-e6870bd84153 file = sled-storage/src/dataset.rs:499
2942024-08-23T21:52:11.188ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:440
2952024-08-23T21:52:11.188ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:357
2962024-08-23T21:52:11.188ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:342
2972024-08-23T21:52:11.188ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2982024-08-23T21:52:11.188ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } path = /scratch/u2_2.vdev
2992024-08-23T21:52:11.198ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_2.vdev file = sled-hardware/src/disk.rs:360
3002024-08-23T21:52:11.198ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool)
3012024-08-23T21:52:11.292ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:307 zpool = ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }
3022024-08-23T21:52:11.299ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:351
3032024-08-23T21:52:11.299ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:353
3042024-08-23T21:52:11.299ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:357
3052024-08-23T21:52:11.299ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:359
3062024-08-23T21:52:11.299ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3072024-08-23T21:52:11.300ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3082024-08-23T21:52:11.300ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:371
3092024-08-23T21:52:11.304ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/5050fe14-0845-40b2-872d-d70a5b2b56be): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
3102024-08-23T21:52:11.336ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3112024-08-23T21:52:11.343ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_0bb3d35c-cbec-445b-9b3a-58337934f8de/crypt/zone file = sled-storage/src/dataset.rs:414
3122024-08-23T21:52:11.474ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_0bb3d35c-cbec-445b-9b3a-58337934f8de file = sled-storage/src/dataset.rs:499
3132024-08-23T21:52:11.483ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:440
3142024-08-23T21:52:11.483ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:357
3152024-08-23T21:52:11.483ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:342
3162024-08-23T21:52:11.483ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3172024-08-23T21:52:11.483ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } path = /scratch/u2_3.vdev
3182024-08-23T21:52:11.493ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_3.vdev file = sled-hardware/src/disk.rs:360
3192024-08-23T21:52:11.493ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 44b92f88-0c07-4732-953e-9cac63715495 (zpool)
3202024-08-23T21:52:11.590ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:307 zpool = ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }
3212024-08-23T21:52:11.599ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:351
3222024-08-23T21:52:11.599ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:353
3232024-08-23T21:52:11.599ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:357
3242024-08-23T21:52:11.599ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:359
3252024-08-23T21:52:11.599ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3262024-08-23T21:52:11.599ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3272024-08-23T21:52:11.599ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:371
3282024-08-23T21:52:11.638ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3292024-08-23T21:52:11.646ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_44b92f88-0c07-4732-953e-9cac63715495/crypt/zone file = sled-storage/src/dataset.rs:414
3302024-08-23T21:52:11.771ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_44b92f88-0c07-4732-953e-9cac63715495 file = sled-storage/src/dataset.rs:499
3312024-08-23T21:52:11.779ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:440
3322024-08-23T21:52:11.779ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:357
3332024-08-23T21:52:11.779ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:342
3342024-08-23T21:52:11.779ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3352024-08-23T21:52:11.779ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } path = /scratch/u2_4.vdev
3362024-08-23T21:52:11.790ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_4.vdev file = sled-hardware/src/disk.rs:360
3372024-08-23T21:52:11.790ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool)
3382024-08-23T21:52:11.884ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:307 zpool = ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }
3392024-08-23T21:52:11.893ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:351
3402024-08-23T21:52:11.893ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:353
3412024-08-23T21:52:11.893ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:357
3422024-08-23T21:52:11.893ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:359
3432024-08-23T21:52:11.893ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3442024-08-23T21:52:11.893ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3452024-08-23T21:52:11.893ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:371
3462024-08-23T21:52:11.930ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3472024-08-23T21:52:11.938ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_44ff68f6-909f-4df8-8e90-d68ca84323b3/crypt/zone file = sled-storage/src/dataset.rs:414
3482024-08-23T21:52:12.065ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_44ff68f6-909f-4df8-8e90-d68ca84323b3 file = sled-storage/src/dataset.rs:499
3492024-08-23T21:52:12.073ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:440
3502024-08-23T21:52:12.073ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:357
3512024-08-23T21:52:12.073ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:342
3522024-08-23T21:52:12.073ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3532024-08-23T21:52:12.073ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } path = /scratch/u2_5.vdev
3542024-08-23T21:52:12.083ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_5.vdev file = sled-hardware/src/disk.rs:360
3552024-08-23T21:52:12.083ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = dad70eae-b83f-4c3a-a32e-2a6a8946803c (zpool)
3562024-08-23T21:52:12.181ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:307 zpool = ZpoolName { id: dad70eae-b83f-4c3a-a32e-2a6a8946803c (zpool), kind: External }
3572024-08-23T21:52:12.190ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:351
3582024-08-23T21:52:12.190ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:353
3592024-08-23T21:52:12.190ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:357
3602024-08-23T21:52:12.190ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:359
3612024-08-23T21:52:12.190ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3622024-08-23T21:52:12.190ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3632024-08-23T21:52:12.190ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:371
3642024-08-23T21:52:12.226ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3652024-08-23T21:52:12.234ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_dad70eae-b83f-4c3a-a32e-2a6a8946803c/crypt/zone file = sled-storage/src/dataset.rs:414
3662024-08-23T21:52:12.306ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/5050fe14-0845-40b2-872d-d70a5b2b56be): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
3672024-08-23T21:52:12.358ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_dad70eae-b83f-4c3a-a32e-2a6a8946803c file = sled-storage/src/dataset.rs:499
3682024-08-23T21:52:12.366ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:440
3692024-08-23T21:52:12.366ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:357
3702024-08-23T21:52:12.366ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:342
3712024-08-23T21:52:12.366ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3722024-08-23T21:52:12.366ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } path = /scratch/u2_6.vdev
3732024-08-23T21:52:12.376ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_6.vdev file = sled-hardware/src/disk.rs:360
3742024-08-23T21:52:12.376ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool)
3752024-08-23T21:52:12.472ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:307 zpool = ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }
3762024-08-23T21:52:12.480ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:351
3772024-08-23T21:52:12.480ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:353
3782024-08-23T21:52:12.480ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:357
3792024-08-23T21:52:12.480ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:359
3802024-08-23T21:52:12.480ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3812024-08-23T21:52:12.480ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3822024-08-23T21:52:12.480ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:371
3832024-08-23T21:52:12.517ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3842024-08-23T21:52:12.526ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_ec50006e-a21c-4820-b439-48f6d6b60c97/crypt/zone file = sled-storage/src/dataset.rs:414
3852024-08-23T21:52:12.658ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_ec50006e-a21c-4820-b439-48f6d6b60c97 file = sled-storage/src/dataset.rs:499
3862024-08-23T21:52:12.666ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:440
3872024-08-23T21:52:12.666ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:357
3882024-08-23T21:52:12.666ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:342
3892024-08-23T21:52:12.666ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3902024-08-23T21:52:12.666ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } path = /scratch/u2_7.vdev
3912024-08-23T21:52:12.676ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_7.vdev file = sled-hardware/src/disk.rs:360
3922024-08-23T21:52:12.676ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = e1a016b1-16ad-48d4-a26f-2e916f51bc80 (zpool)
3932024-08-23T21:52:12.776ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:307 zpool = ZpoolName { id: e1a016b1-16ad-48d4-a26f-2e916f51bc80 (zpool), kind: External }
3942024-08-23T21:52:12.784ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:351
3952024-08-23T21:52:12.784ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:353
3962024-08-23T21:52:12.784ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:357
3972024-08-23T21:52:12.784ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:359
3982024-08-23T21:52:12.784ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
3992024-08-23T21:52:12.784ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4002024-08-23T21:52:12.784ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:371
4012024-08-23T21:52:12.820ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4022024-08-23T21:52:12.828ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_e1a016b1-16ad-48d4-a26f-2e916f51bc80/crypt/zone file = sled-storage/src/dataset.rs:414
4032024-08-23T21:52:12.953ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_e1a016b1-16ad-48d4-a26f-2e916f51bc80 file = sled-storage/src/dataset.rs:499
4042024-08-23T21:52:12.961ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:440
4052024-08-23T21:52:12.961ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:357
4062024-08-23T21:52:12.961ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:342
4072024-08-23T21:52:12.961ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
4082024-08-23T21:52:12.961ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } path = /scratch/u2_8.vdev
4092024-08-23T21:52:12.971ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_8.vdev file = sled-hardware/src/disk.rs:360
4102024-08-23T21:52:12.971ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 7ac03450-9a27-4ec4-9669-9eb798da4419 (zpool)
4112024-08-23T21:52:13.067ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:307 zpool = ZpoolName { id: 7ac03450-9a27-4ec4-9669-9eb798da4419 (zpool), kind: External }
4122024-08-23T21:52:13.075ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:351
4132024-08-23T21:52:13.075ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:353
4142024-08-23T21:52:13.075ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:357
4152024-08-23T21:52:13.075ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:359
4162024-08-23T21:52:13.075ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4172024-08-23T21:52:13.075ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4182024-08-23T21:52:13.075ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:371
4192024-08-23T21:52:13.107ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4202024-08-23T21:52:13.114ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_7ac03450-9a27-4ec4-9669-9eb798da4419/crypt/zone file = sled-storage/src/dataset.rs:414
4212024-08-23T21:52:13.238ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_7ac03450-9a27-4ec4-9669-9eb798da4419 file = sled-storage/src/dataset.rs:499
4222024-08-23T21:52:13.246ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:440
4232024-08-23T21:52:13.246ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:357
4242024-08-23T21:52:13.246ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4252024-08-23T21:52:13.246ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 156808b0-3956-455e-a8e1-2217e0afca54 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: fe7217f3-1112-4ad5-a4fc-b58965332aab (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: dad70eae-b83f-4c3a-a32e-2a6a8946803c (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: e1a016b1-16ad-48d4-a26f-2e916f51bc80 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 7ac03450-9a27-4ec4-9669-9eb798da4419 (zpool), kind: External } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
4262024-08-23T21:52:13.246ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
4272024-08-23T21:52:13.246ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
4282024-08-23T21:52:13.247ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4292024-08-23T21:52:13.247ZINFOSledAgent: physical disks ensure: Updated storage file = sled-agent/src/sled_agent.rs:837 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
4302024-08-23T21:52:13.247ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
4312024-08-23T21:52:13.247ZINFOSledAgent: physical disks ensure: Propagating new generation of disks file = sled-agent/src/sled_agent.rs:863 generation = Generation(1) sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
4322024-08-23T21:52:13.247ZINFOSledAgent: physical disks ensure: Updated storage monitor file = sled-agent/src/sled_agent.rs:868 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
4332024-08-23T21:52:13.247ZINFOSledAgent: physical disks ensure: Updated zone bundler file = sled-agent/src/sled_agent.rs:873 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
4342024-08-23T21:52:13.247ZINFOSledAgent (ProbeManager): use_only_these_disks: Processing new request file = sled-agent/src/probe_manager.rs:121 gen = Generation(1) sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
4352024-08-23T21:52:13.247ZINFOSledAgent: physical disks ensure: Updated probes file = sled-agent/src/sled_agent.rs:879 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
4362024-08-23T21:52:13.247ZINFOSledAgent (InstanceManager): use_only_these_disks: Processing new request file = sled-agent/src/instance_manager.rs:798 gen = Generation(1)
4372024-08-23T21:52:13.247ZINFOSledAgent: physical disks ensure: Updated instances file = sled-agent/src/sled_agent.rs:884 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
4382024-08-23T21:52:13.247ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 3682177 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59727 req_id = 8c2ceaac-a797-4d80-9c4f-a5ce33a38949 response_code = 200 uri = /omicron-physical-disks
4392024-08-23T21:52:13.278ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(2), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
4402024-08-23T21:52:13.278ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:36907
4412024-08-23T21:52:13.279ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
4422024-08-23T21:52:13.279ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4432024-08-23T21:52:13.308ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/5050fe14-0845-40b2-872d-d70a5b2b56be): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
4442024-08-23T21:52:13.332ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
4452024-08-23T21:52:13.332ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4462024-08-23T21:52:13.382ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
4472024-08-23T21:52:13.382ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
4482024-08-23T21:52:13.431ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
4492024-08-23T21:52:13.431ZINFOSledAgent (ServiceManager): No ledger in /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:188
4502024-08-23T21:52:13.431ZINFOSledAgent (ServiceManager): No ledger in /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:188
4512024-08-23T21:52:13.431ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
4522024-08-23T21:52:13.475ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
4532024-08-23T21:52:13.475ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
4542024-08-23T21:52:13.475ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
4552024-08-23T21:52:13.525ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_bd1e844a-019f-4d2c-af1c-ec9d27991436 file = illumos-utils/src/zone.rs:316
4562024-08-23T21:52:13.529ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_60c8c02e-6e6f-4e1a-a162-ca6470555313 file = illumos-utils/src/zone.rs:316
4572024-08-23T21:52:13.532ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_07617dac-1d06-4704-81b5-9345e9dd66a9 file = illumos-utils/src/zone.rs:316
4582024-08-23T21:52:13.545ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_bd1e844a-019f-4d2c-af1c-ec9d27991436 file = illumos-utils/src/zone.rs:354
4592024-08-23T21:52:13.549ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_60c8c02e-6e6f-4e1a-a162-ca6470555313 file = illumos-utils/src/zone.rs:354
4602024-08-23T21:52:13.556ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_07617dac-1d06-4704-81b5-9345e9dd66a9 file = illumos-utils/src/zone.rs:354
4612024-08-23T21:52:14.310ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/5050fe14-0845-40b2-872d-d70a5b2b56be): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
4622024-08-23T21:52:14.344ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 22.208073579s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
4632024-08-23T21:52:15.042ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_bd1e844a-019f-4d2c-af1c-ec9d27991436: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService1'/> <propval type="astring" name="gateway" value='fd00:1122:3344:1::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:1::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:1::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:1::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4642024-08-23T21:52:15.042ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:1::, width: 64 }
4652024-08-23T21:52:15.042ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_bd1e844a-019f-4d2c-af1c-ec9d27991436 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_bd1e844a-019f-4d2c-af1c-ec9d27991436
4662024-08-23T21:52:15.525ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_60c8c02e-6e6f-4e1a-a162-ca6470555313: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService2'/> <propval type="astring" name="gateway" value='fd00:1122:3344:2::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:2::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:2::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:2::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4672024-08-23T21:52:15.525ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:2::, width: 64 }
4682024-08-23T21:52:16.007ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_07617dac-1d06-4704-81b5-9345e9dd66a9: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService3'/> <propval type="astring" name="gateway" value='fd00:1122:3344:3::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:3::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:3::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:3::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4692024-08-23T21:52:16.016ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_bd1e844a-019f-4d2c-af1c-ec9d27991436") failed: Property not found. retry in 72.033559ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_bd1e844a-019f-4d2c-af1c-ec9d27991436
4702024-08-23T21:52:16.016ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_60c8c02e-6e6f-4e1a-a162-ca6470555313 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_60c8c02e-6e6f-4e1a-a162-ca6470555313
4712024-08-23T21:52:16.020ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:3::, width: 64 }
4722024-08-23T21:52:16.020ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_07617dac-1d06-4704-81b5-9345e9dd66a9 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_07617dac-1d06-4704-81b5-9345e9dd66a9
4732024-08-23T21:52:16.098ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_bd1e844a-019f-4d2c-af1c-ec9d27991436") failed: Property not found. retry in 118.675481ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_bd1e844a-019f-4d2c-af1c-ec9d27991436
4742024-08-23T21:52:16.144ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_60c8c02e-6e6f-4e1a-a162-ca6470555313") failed: Property not found. retry in 73.014811ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_60c8c02e-6e6f-4e1a-a162-ca6470555313
4752024-08-23T21:52:16.151ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_07617dac-1d06-4704-81b5-9345e9dd66a9") failed: Property not found. retry in 57.98144ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_07617dac-1d06-4704-81b5-9345e9dd66a9
4762024-08-23T21:52:16.286ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_07617dac-1d06-4704-81b5-9345e9dd66a9") failed: Property not found. retry in 100.634493ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_07617dac-1d06-4704-81b5-9345e9dd66a9
4772024-08-23T21:52:16.338ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_bd1e844a-019f-4d2c-af1c-ec9d27991436") failed: Property not found. retry in 200.789185ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_bd1e844a-019f-4d2c-af1c-ec9d27991436
4782024-08-23T21:52:16.363ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_60c8c02e-6e6f-4e1a-a162-ca6470555313") failed: Property not found. retry in 89.14496ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_60c8c02e-6e6f-4e1a-a162-ca6470555313
4792024-08-23T21:52:16.461ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_07617dac-1d06-4704-81b5-9345e9dd66a9") failed: Property not found. retry in 123.852131ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_07617dac-1d06-4704-81b5-9345e9dd66a9
4802024-08-23T21:52:16.565ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_60c8c02e-6e6f-4e1a-a162-ca6470555313") failed: Property not found. retry in 224.28907ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_60c8c02e-6e6f-4e1a-a162-ca6470555313
4812024-08-23T21:52:16.600ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_bd1e844a-019f-4d2c-af1c-ec9d27991436") failed: Property not found. retry in 414.527347ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_bd1e844a-019f-4d2c-af1c-ec9d27991436
4822024-08-23T21:52:16.614ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_07617dac-1d06-4704-81b5-9345e9dd66a9") failed: Property not found. retry in 432.023261ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_07617dac-1d06-4704-81b5-9345e9dd66a9
4832024-08-23T21:52:16.806ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_60c8c02e-6e6f-4e1a-a162-ca6470555313") failed: Property not found. retry in 538.980145ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_60c8c02e-6e6f-4e1a-a162-ca6470555313
4842024-08-23T21:52:17.023ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_bd1e844a-019f-4d2c-af1c-ec9d27991436") failed: Property not found. retry in 1.168075977s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_bd1e844a-019f-4d2c-af1c-ec9d27991436
4852024-08-23T21:52:17.055ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_07617dac-1d06-4704-81b5-9345e9dd66a9") failed: Property not found. retry in 490.537997ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_07617dac-1d06-4704-81b5-9345e9dd66a9
4862024-08-23T21:52:17.355ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_60c8c02e-6e6f-4e1a-a162-ca6470555313") failed: Property not found. retry in 1.129984222s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_60c8c02e-6e6f-4e1a-a162-ca6470555313
4872024-08-23T21:52:17.557ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_07617dac-1d06-4704-81b5-9345e9dd66a9") failed: Property not found. retry in 885.435162ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_07617dac-1d06-4704-81b5-9345e9dd66a9
4882024-08-23T21:52:18.200ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_bd1e844a-019f-4d2c-af1c-ec9d27991436") failed: Property not found. retry in 1.119804346s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_bd1e844a-019f-4d2c-af1c-ec9d27991436
4892024-08-23T21:52:18.451ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_07617dac-1d06-4704-81b5-9345e9dd66a9") failed: Property not found. retry in 726.686972ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_07617dac-1d06-4704-81b5-9345e9dd66a9
4902024-08-23T21:52:18.493ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_60c8c02e-6e6f-4e1a-a162-ca6470555313") failed: Property not found. retry in 1.335967019s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_60c8c02e-6e6f-4e1a-a162-ca6470555313
4912024-08-23T21:52:19.124ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists - OK file = sled-agent/src/services.rs:4206
4922024-08-23T21:52:19.124ZINFOSledAgent (ServiceManager): Ensuring there is a default route file = sled-agent/src/services.rs:4216 gateway = fd00:1122:3344:101::1
4932024-08-23T21:52:19.130ZINFOSledAgent (ServiceManager): configuring dendrite service file = sled-agent/src/services.rs:4295
4942024-08-23T21:52:19.186ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_07617dac-1d06-4704-81b5-9345e9dd66a9") failed: Property not found. retry in 1.424185158s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_07617dac-1d06-4704-81b5-9345e9dd66a9
4952024-08-23T21:52:19.252ZINFOSledAgent (ServiceManager): refreshed dendrite service with new configuration file = sled-agent/src/services.rs:4343
4962024-08-23T21:52:19.252ZINFOSledAgent (ServiceManager): configuring lldp service file = sled-agent/src/services.rs:4370
4972024-08-23T21:52:19.290ZINFOSledAgent (ServiceManager): refreshed lldpd service with new configuration file = sled-agent/src/services.rs:4383
4982024-08-23T21:52:19.290ZINFOSledAgent (ServiceManager): configuring MGS service file = sled-agent/src/services.rs:4247
4992024-08-23T21:52:19.329ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_bd1e844a-019f-4d2c-af1c-ec9d27991436") failed: Property not found. retry in 952.541041ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_bd1e844a-019f-4d2c-af1c-ec9d27991436
5002024-08-23T21:52:19.345ZINFOSledAgent (ServiceManager): refreshed MGS service with new configuration file = sled-agent/src/services.rs:4289
5012024-08-23T21:52:19.345ZINFOSledAgent (ServiceManager): configuring wicketd file = sled-agent/src/services.rs:4350 rack_subnet = fd00:1122:3344::
5022024-08-23T21:52:19.365ZINFOSledAgent (ServiceManager): refreshed wicketd service with new configuration file = sled-agent/src/services.rs:4361
5032024-08-23T21:52:19.365ZINFOSledAgent (ServiceManager): configuring mgd service file = sled-agent/src/services.rs:4403
5042024-08-23T21:52:19.449ZINFOSledAgent (ServiceManager): refreshed mgd service with new configuration file = sled-agent/src/services.rs:4435
5052024-08-23T21:52:19.449ZINFOSledAgent (ServiceManager): configuring mg-ddm service file = sled-agent/src/services.rs:4441
5062024-08-23T21:52:19.558ZINFOSledAgent (ServiceManager): refreshed mg-ddm service with new configuration file = sled-agent/src/services.rs:4482
5072024-08-23T21:52:19.558ZINFOSledAgent (ServiceManager): Determining physical location of our switch zone at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:366
5082024-08-23T21:52:19.591ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/5050fe14-0845-40b2-872d-d70a5b2b56be): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:571 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
5092024-08-23T21:52:19.591ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 45.859982ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/bootstrap/early_networking.rs:385
5102024-08-23T21:52:19.638ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 75.85804ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/bootstrap/early_networking.rs:385
5112024-08-23T21:52:19.714ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 185.599618ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/bootstrap/early_networking.rs:385
5122024-08-23T21:52:19.840ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_60c8c02e-6e6f-4e1a-a162-ca6470555313") failed: Property not found. retry in 1.142735352s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_60c8c02e-6e6f-4e1a-a162-ca6470555313
5132024-08-23T21:52:19.901ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 321.444765ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/bootstrap/early_networking.rs:385
5142024-08-23T21:52:20.199ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::2]:43133
5152024-08-23T21:52:20.200ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 70 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::2]:43133 req_id = 5fdb6aac-458d-48b1-aafa-33c6a6447aff response_code = 200 uri = /sled-identifiers
5162024-08-23T21:52:20.224ZINFOSledAgent (ServiceManager): Initializing 1 Uplinks on Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:414
5172024-08-23T21:52:20.259ZINFOSledAgent (ServiceManager): Building Port Configuration file = sled-agent/src/bootstrap/early_networking.rs:676
5182024-08-23T21:52:20.259ZINFOSledAgent (ServiceManager): Checking dendrite uptime file = sled-agent/src/bootstrap/early_networking.rs:714
5192024-08-23T21:52:20.260ZINFOSledAgent (ServiceManager): Dendrite online file = sled-agent/src/bootstrap/early_networking.rs:717 uptime = 0
5202024-08-23T21:52:20.260ZINFOSledAgent (ServiceManager): Configuring default uplink on switch config = PortSettings {\n links: {\n "0": LinkSettings {\n addrs: [\n 10.151.1.174,\n ],\n params: LinkCreate {\n autoneg: false,\n fec: None,\n kr: false,\n lane: Some(\n LinkId(\n 0,\n ),\n ),\n speed: Speed40G,\n },\n },\n },\n} file = sled-agent/src/bootstrap/early_networking.rs:436
5212024-08-23T21:52:20.301ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_bd1e844a-019f-4d2c-af1c-ec9d27991436") failed: Property not found. retry in 561.701388ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_bd1e844a-019f-4d2c-af1c-ec9d27991436
5222024-08-23T21:52:20.303ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
5232024-08-23T21:52:20.356ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:406
5242024-08-23T21:52:20.357ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5252024-08-23T21:52:20.357ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5262024-08-23T21:52:20.357ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5272024-08-23T21:52:20.357ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5282024-08-23T21:52:20.357ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5292024-08-23T21:52:20.357ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5302024-08-23T21:52:20.357ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5312024-08-23T21:52:20.357ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5322024-08-23T21:52:20.357ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5332024-08-23T21:52:20.357ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5342024-08-23T21:52:20.357ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5352024-08-23T21:52:20.357ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5362024-08-23T21:52:20.357ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5372024-08-23T21:52:20.357ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5382024-08-23T21:52:20.357ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5392024-08-23T21:52:20.357ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5402024-08-23T21:52:20.357ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5412024-08-23T21:52:20.357ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5422024-08-23T21:52:20.357ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5432024-08-23T21:52:20.357ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5442024-08-23T21:52:20.357ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5452024-08-23T21:52:20.357ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5462024-08-23T21:52:20.619ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_07617dac-1d06-4704-81b5-9345e9dd66a9") failed: Property not found. retry in 1.130143405s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_07617dac-1d06-4704-81b5-9345e9dd66a9
5472024-08-23T21:52:21.594ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/5050fe14-0845-40b2-872d-d70a5b2b56be): error trying to connect: dns error: no record found for Query { name: Name("_nexus._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } file = sled-agent/src/nexus.rs:571 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
5482024-08-23T21:52:21.770ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3245 zone = oxz_internal_dns_07617dac-1d06-4704-81b5-9345e9dd66a9
5492024-08-23T21:52:21.770ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3245 zone = oxz_internal_dns_bd1e844a-019f-4d2c-af1c-ec9d27991436
5502024-08-23T21:52:21.770ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3245 zone = oxz_internal_dns_60c8c02e-6e6f-4e1a-a162-ca6470555313
5512024-08-23T21:52:21.770ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5522024-08-23T21:52:21.770ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5532024-08-23T21:52:21.771ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8492339 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:36907 req_id = fe2f1be5-f1d2-4ca8-aa7f-fb678c3ec123 response_code = 204 uri = /omicron-zones
5542024-08-23T21:52:21.773ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:1::1]:5353 file = sled-agent/src/rack_setup/service.rs:557
5552024-08-23T21:52:21.825ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:2::1]:5353 file = sled-agent/src/rack_setup/service.rs:557
5562024-08-23T21:52:21.870ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:3::1]:5353 file = sled-agent/src/rack_setup/service.rs:557
5572024-08-23T21:52:21.917ZINFOSledAgent (RSS): Configured all DNS servers file = sled-agent/src/rack_setup/service.rs:593
5582024-08-23T21:52:21.917ZINFOSledAgent (RSS): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
5592024-08-23T21:52:21.947ZINFOSledAgent (RSS): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
5602024-08-23T21:52:21.948ZINFOSledAgent (RSS): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
5612024-08-23T21:52:21.948ZINFOSledAgent (RSS): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
5622024-08-23T21:52:21.977ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(3), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
5632024-08-23T21:52:21.977ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:51899
5642024-08-23T21:52:21.977ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
5652024-08-23T21:52:21.977ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5662024-08-23T21:52:21.999ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
5672024-08-23T21:52:21.999ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5682024-08-23T21:52:22.020ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
5692024-08-23T21:52:22.020ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
5702024-08-23T21:52:22.038ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
5712024-08-23T21:52:22.038ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
5722024-08-23T21:52:22.038ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
5732024-08-23T21:52:22.038ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
5742024-08-23T21:52:22.049ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
5752024-08-23T21:52:22.083ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
5762024-08-23T21:52:22.084ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
5772024-08-23T21:52:22.084ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
5782024-08-23T21:52:22.114ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte0", ip: 172.30.3.5, mac: MacAddr6([168, 64, 37, 255, 151, 153]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.3.1 } }) }
5792024-08-23T21:52:22.114ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1235 zone_type = boundary_ntp
5802024-08-23T21:52:22.114ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:83
5812024-08-23T21:52:22.114ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:90
5822024-08-23T21:52:22.115ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:118
5832024-08-23T21:52:22.116ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
5842024-08-23T21:52:22.139ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_ntp_d738a4fe-561c-4114-8fad-21f02e2dba0e file = illumos-utils/src/zone.rs:316
5852024-08-23T21:52:22.156ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_ntp_d738a4fe-561c-4114-8fad-21f02e2dba0e file = illumos-utils/src/zone.rs:354
5862024-08-23T21:52:22.756ZINFOSledAgent (ServiceManager): Profile for oxz_ntp_d738a4fe-561c-4114-8fad-21f02e2dba0e: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService4'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/chrony-setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="allow" value='fd00:1122:3344:100::/56'/> <propval type="boolean" name="boundary" value='true'/> <propval type="astring" name="boundary_pool" value='boundary-ntp.control-plane.oxide.internal'/> <propval type="astring" name="server" value='0.pool.ntp.org'/> </property_group> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="1.1.1.1" /> <value_node value="9.9.9.9" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/ntp"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte0'/> <propval type="astring" name="gateway" value='172.30.3.1'/> <propval type="astring" name="ip" value='172.30.3.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
5872024-08-23T21:52:22.756ZINFOSledAgent (ServiceManager): Booting oxz_ntp_d738a4fe-561c-4114-8fad-21f02e2dba0e zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_ntp_d738a4fe-561c-4114-8fad-21f02e2dba0e
5882024-08-23T21:52:22.884ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_d738a4fe-561c-4114-8fad-21f02e2dba0e") failed: Property not found. retry in 55.297264ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_d738a4fe-561c-4114-8fad-21f02e2dba0e
5892024-08-23T21:52:22.949ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_d738a4fe-561c-4114-8fad-21f02e2dba0e") failed: Property not found. retry in 106.004364ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_d738a4fe-561c-4114-8fad-21f02e2dba0e
5902024-08-23T21:52:23.089ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_d738a4fe-561c-4114-8fad-21f02e2dba0e") failed: Property not found. retry in 279.248715ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_d738a4fe-561c-4114-8fad-21f02e2dba0e
5912024-08-23T21:52:23.380ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_d738a4fe-561c-4114-8fad-21f02e2dba0e") failed: Property not found. retry in 320.504776ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_d738a4fe-561c-4114-8fad-21f02e2dba0e
5922024-08-23T21:52:23.712ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_d738a4fe-561c-4114-8fad-21f02e2dba0e") failed: Property not found. retry in 877.343701ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_d738a4fe-561c-4114-8fad-21f02e2dba0e
5932024-08-23T21:52:24.599ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_d738a4fe-561c-4114-8fad-21f02e2dba0e") failed: Property not found. retry in 1.465485883s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_d738a4fe-561c-4114-8fad-21f02e2dba0e
5942024-08-23T21:52:26.074ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_d738a4fe-561c-4114-8fad-21f02e2dba0e") failed: Property not found. retry in 911.531742ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_d738a4fe-561c-4114-8fad-21f02e2dba0e
5952024-08-23T21:52:26.995ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_d738a4fe-561c-4114-8fad-21f02e2dba0e") failed: Property not found. retry in 790.18764ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_d738a4fe-561c-4114-8fad-21f02e2dba0e
5962024-08-23T21:52:27.808ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3245 zone = oxz_ntp_d738a4fe-561c-4114-8fad-21f02e2dba0e
5972024-08-23T21:52:27.808ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5982024-08-23T21:52:27.809ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5992024-08-23T21:52:27.809ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 5832118 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:51899 req_id = c8824799-7ed1-4752-8215-0e8debe6a951 response_code = 204 uri = /omicron-zones
6002024-08-23T21:52:27.811ZINFOSledAgent (RSS): Waiting for rack time synchronization file = sled-agent/src/rack_setup/service.rs:634
6012024-08-23T21:52:27.842ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:614
6022024-08-23T21:52:27.842ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:51874
6032024-08-23T21:52:27.850ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_d738a4fe-561c-4114-8fad-21f02e2dba0e': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3847
6042024-08-23T21:52:27.851ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 8776 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:51874 req_id = 7b2de4b0-dfe9-4471-b5ef-746a673efa49 response_code = 500 uri = /timesync
6052024-08-23T21:52:27.852ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:664
6062024-08-23T21:52:27.973ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:614
6072024-08-23T21:52:27.973ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:56779
6082024-08-23T21:52:31.982ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_d738a4fe-561c-4114-8fad-21f02e2dba0e': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3847
6092024-08-23T21:52:31.982ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 4008874 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:56779 req_id = 164406a0-0670-47a7-a8a5-154e779aa623 response_code = 500 uri = /timesync
6102024-08-23T21:52:31.986ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:664
6112024-08-23T21:52:32.178ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:614
6122024-08-23T21:52:32.178ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:40734
6132024-08-23T21:52:32.187ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_d738a4fe-561c-4114-8fad-21f02e2dba0e': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3847
6142024-08-23T21:52:32.187ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 9062 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:40734 req_id = ec00d723-1dcf-4ea8-bacb-22997d74046f response_code = 500 uri = /timesync
6152024-08-23T21:52:32.189ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:664
6162024-08-23T21:52:32.400ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:614
6172024-08-23T21:52:32.400ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:49770
6182024-08-23T21:52:32.408ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_d738a4fe-561c-4114-8fad-21f02e2dba0e': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3847
6192024-08-23T21:52:32.408ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 8172 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:49770 req_id = 367c6183-745f-483a-99e1-dc161093ef63 response_code = 500 uri = /timesync
6202024-08-23T21:52:32.410ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:664
6212024-08-23T21:52:32.686ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:614
6222024-08-23T21:52:32.686ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:36231
6232024-08-23T21:52:33.696ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1009145 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:36231 req_id = 9287865d-9d0a-429b-94cb-6586457e6327 response_code = 200 uri = /timesync
6242024-08-23T21:52:33.697ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:642
6252024-08-23T21:52:33.697ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:664
6262024-08-23T21:52:33.937ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:614
6272024-08-23T21:52:33.937ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:62256
6282024-08-23T21:52:33.946ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8300 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:62256 req_id = 15d430c2-53b7-4b2d-9e9c-966ed1c5108f response_code = 200 uri = /timesync
6292024-08-23T21:52:33.949ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:642
6302024-08-23T21:52:33.949ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:664
6312024-08-23T21:52:34.247ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:614
6322024-08-23T21:52:34.247ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:34956
6332024-08-23T21:52:34.256ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8653 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:34956 req_id = 1fb21413-d749-4e92-b544-94bb98d57a49 response_code = 200 uri = /timesync
6342024-08-23T21:52:34.258ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:642
6352024-08-23T21:52:34.258ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:664
6362024-08-23T21:52:34.551ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:614
6372024-08-23T21:52:34.551ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:39525
6382024-08-23T21:52:34.559ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8162 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:39525 req_id = 5a3a608c-4632-4984-abb2-14c54268ca2d response_code = 200 uri = /timesync
6392024-08-23T21:52:34.561ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:642
6402024-08-23T21:52:34.561ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:664
6412024-08-23T21:52:34.780ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:614
6422024-08-23T21:52:34.781ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:34858
6432024-08-23T21:52:34.788ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7763 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:34858 req_id = 231a43c9-9900-4fa4-b52c-d8f578b2ee69 response_code = 200 uri = /timesync
6442024-08-23T21:52:34.791ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:642
6452024-08-23T21:52:34.791ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:664
6462024-08-23T21:52:35.234ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:614
6472024-08-23T21:52:35.234ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:60676
6482024-08-23T21:52:35.241ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7320 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:60676 req_id = 4018e901-e743-4f5a-a0a0-adadc5eea5f7 response_code = 200 uri = /timesync
6492024-08-23T21:52:35.245ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:642
6502024-08-23T21:52:35.245ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:664
6512024-08-23T21:52:35.356ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:614
6522024-08-23T21:52:35.356ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:38029
6532024-08-23T21:52:35.376ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 19517 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:38029 req_id = 868b043e-9838-486c-aaa0-79dacd8da26d response_code = 200 uri = /timesync
6542024-08-23T21:52:35.379ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: true, ref_id: 2273734522, ip_addr: 135.134.111.122, stratum: 3, ref_time: 1724449955.0739956, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:642
6552024-08-23T21:52:35.379ZINFOSledAgent (RSS): Finished setting up Internal DNS and NTP file = sled-agent/src/rack_setup/service.rs:1266
6562024-08-23T21:52:35.410ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
6572024-08-23T21:52:35.410ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:49912
6582024-08-23T21:52:35.410ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
6592024-08-23T21:52:35.410ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6602024-08-23T21:52:35.430ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
6612024-08-23T21:52:35.430ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6622024-08-23T21:52:35.448ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
6632024-08-23T21:52:35.449ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
6642024-08-23T21:52:35.466ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
6652024-08-23T21:52:35.466ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6662024-08-23T21:52:35.518ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
6672024-08-23T21:52:35.518ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6682024-08-23T21:52:35.570ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
6692024-08-23T21:52:35.570ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6702024-08-23T21:52:35.622ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
6712024-08-23T21:52:35.622ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6722024-08-23T21:52:35.674ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
6732024-08-23T21:52:35.674ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6742024-08-23T21:52:35.724ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
6752024-08-23T21:52:35.724ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
6762024-08-23T21:52:35.724ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
6772024-08-23T21:52:35.725ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
6782024-08-23T21:52:35.756ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
6792024-08-23T21:52:35.756ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
6802024-08-23T21:52:35.756ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
6812024-08-23T21:52:35.756ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
6822024-08-23T21:52:35.756ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
6832024-08-23T21:52:35.756ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657", TimeNotSynchronized), ("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb", TimeNotSynchronized), ("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd", TimeNotSynchronized), ("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8", TimeNotSynchronized), ("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 346390 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49912 req_id = a3166e82-34f4-48ba-87b7-307d8c177a23 response_code = 500 uri = /omicron-zones
6842024-08-23T21:52:35.757ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "a3166e82-34f4-48ba-87b7-307d8c177a23", "content-length": "124", "date": "Fri, 23 Aug 2024 21:52:35 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "a3166e82-34f4-48ba-87b7-307d8c177a23" } file = sled-agent/src/rack_setup/service.rs:477 retry_after = 100.509801ms sled_address = [fd00:1122:3344:101::1]:12345
6852024-08-23T21:52:35.858ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
6862024-08-23T21:52:35.858ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
6872024-08-23T21:52:35.858ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6882024-08-23T21:52:35.876ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
6892024-08-23T21:52:35.876ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6902024-08-23T21:52:35.895ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
6912024-08-23T21:52:35.895ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
6922024-08-23T21:52:35.912ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
6932024-08-23T21:52:35.912ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6942024-08-23T21:52:35.930ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
6952024-08-23T21:52:35.930ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6962024-08-23T21:52:35.948ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
6972024-08-23T21:52:35.949ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6982024-08-23T21:52:35.966ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
6992024-08-23T21:52:35.966ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7002024-08-23T21:52:35.985ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
7012024-08-23T21:52:35.985ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7022024-08-23T21:52:36.003ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
7032024-08-23T21:52:36.003ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
7042024-08-23T21:52:36.003ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
7052024-08-23T21:52:36.003ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
7062024-08-23T21:52:36.025ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
7072024-08-23T21:52:36.025ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
7082024-08-23T21:52:36.025ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
7092024-08-23T21:52:36.025ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
7102024-08-23T21:52:36.025ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
7112024-08-23T21:52:36.025ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855", TimeNotSynchronized), ("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd", TimeNotSynchronized), ("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb", TimeNotSynchronized), ("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8", TimeNotSynchronized), ("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 177566 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49912 req_id = 5ade7c4a-df27-4abd-9231-9bb81b8afecc response_code = 500 uri = /omicron-zones
7122024-08-23T21:52:36.026ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "5ade7c4a-df27-4abd-9231-9bb81b8afecc", "content-length": "124", "date": "Fri, 23 Aug 2024 21:52:35 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "5ade7c4a-df27-4abd-9231-9bb81b8afecc" } file = sled-agent/src/rack_setup/service.rs:477 retry_after = 138.350117ms sled_address = [fd00:1122:3344:101::1]:12345
7132024-08-23T21:52:36.165ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
7142024-08-23T21:52:36.165ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
7152024-08-23T21:52:36.165ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7162024-08-23T21:52:36.184ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
7172024-08-23T21:52:36.184ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7182024-08-23T21:52:36.202ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
7192024-08-23T21:52:36.202ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
7202024-08-23T21:52:36.220ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
7212024-08-23T21:52:36.220ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7222024-08-23T21:52:36.238ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
7232024-08-23T21:52:36.238ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7242024-08-23T21:52:36.256ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
7252024-08-23T21:52:36.256ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7262024-08-23T21:52:36.274ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
7272024-08-23T21:52:36.274ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7282024-08-23T21:52:36.291ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
7292024-08-23T21:52:36.292ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7302024-08-23T21:52:36.310ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
7312024-08-23T21:52:36.310ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
7322024-08-23T21:52:36.310ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
7332024-08-23T21:52:36.310ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
7342024-08-23T21:52:36.344ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
7352024-08-23T21:52:36.344ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
7362024-08-23T21:52:36.344ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
7372024-08-23T21:52:36.344ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
7382024-08-23T21:52:36.344ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
7392024-08-23T21:52:36.344ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb", TimeNotSynchronized), ("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855", TimeNotSynchronized), ("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd", TimeNotSynchronized), ("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657", TimeNotSynchronized), ("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 178939 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49912 req_id = 6c2802de-e219-4215-a750-ed0b5f279fe9 response_code = 500 uri = /omicron-zones
7402024-08-23T21:52:36.344ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "6c2802de-e219-4215-a750-ed0b5f279fe9", "content-length": "124", "date": "Fri, 23 Aug 2024 21:52:35 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "6c2802de-e219-4215-a750-ed0b5f279fe9" } file = sled-agent/src/rack_setup/service.rs:477 retry_after = 200.666262ms sled_address = [fd00:1122:3344:101::1]:12345
7412024-08-23T21:52:36.546ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
7422024-08-23T21:52:36.546ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
7432024-08-23T21:52:36.546ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7442024-08-23T21:52:36.566ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
7452024-08-23T21:52:36.566ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7462024-08-23T21:52:36.585ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
7472024-08-23T21:52:36.585ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
7482024-08-23T21:52:36.604ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
7492024-08-23T21:52:36.604ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7502024-08-23T21:52:36.622ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
7512024-08-23T21:52:36.622ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7522024-08-23T21:52:36.641ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
7532024-08-23T21:52:36.641ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7542024-08-23T21:52:36.659ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
7552024-08-23T21:52:36.659ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7562024-08-23T21:52:36.676ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
7572024-08-23T21:52:36.677ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7582024-08-23T21:52:36.694ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
7592024-08-23T21:52:36.694ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
7602024-08-23T21:52:36.695ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
7612024-08-23T21:52:36.695ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
7622024-08-23T21:52:36.727ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
7632024-08-23T21:52:36.727ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
7642024-08-23T21:52:36.727ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
7652024-08-23T21:52:36.727ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
7662024-08-23T21:52:36.727ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
7672024-08-23T21:52:36.727ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb", TimeNotSynchronized), ("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd", TimeNotSynchronized), ("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657", TimeNotSynchronized), ("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8", TimeNotSynchronized), ("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 180821 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49912 req_id = aacca9b2-e399-48ff-8574-95a61894f833 response_code = 500 uri = /omicron-zones
7682024-08-23T21:52:36.727ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "aacca9b2-e399-48ff-8574-95a61894f833", "content-length": "124", "date": "Fri, 23 Aug 2024 21:52:36 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "aacca9b2-e399-48ff-8574-95a61894f833" } file = sled-agent/src/rack_setup/service.rs:477 retry_after = 97.016408ms sled_address = [fd00:1122:3344:101::1]:12345
7692024-08-23T21:52:36.825ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
7702024-08-23T21:52:36.825ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
7712024-08-23T21:52:36.825ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7722024-08-23T21:52:36.845ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
7732024-08-23T21:52:36.845ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7742024-08-23T21:52:36.861ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
7752024-08-23T21:52:36.861ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
7762024-08-23T21:52:36.876ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
7772024-08-23T21:52:36.877ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7782024-08-23T21:52:36.895ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
7792024-08-23T21:52:36.895ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7802024-08-23T21:52:36.911ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
7812024-08-23T21:52:36.911ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7822024-08-23T21:52:36.927ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
7832024-08-23T21:52:36.927ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7842024-08-23T21:52:36.944ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
7852024-08-23T21:52:36.944ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7862024-08-23T21:52:36.964ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
7872024-08-23T21:52:36.964ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
7882024-08-23T21:52:36.964ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
7892024-08-23T21:52:36.964ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
7902024-08-23T21:52:36.995ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
7912024-08-23T21:52:36.995ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
7922024-08-23T21:52:36.995ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
7932024-08-23T21:52:36.995ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
7942024-08-23T21:52:36.995ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
7952024-08-23T21:52:36.995ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657", TimeNotSynchronized), ("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb", TimeNotSynchronized), ("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8", TimeNotSynchronized), ("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855", TimeNotSynchronized), ("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 170466 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49912 req_id = ea5ac5f2-2d52-49ae-b911-01b8d8e8de58 response_code = 500 uri = /omicron-zones
7962024-08-23T21:52:36.995ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "ea5ac5f2-2d52-49ae-b911-01b8d8e8de58", "content-length": "124", "date": "Fri, 23 Aug 2024 21:52:36 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "ea5ac5f2-2d52-49ae-b911-01b8d8e8de58" } file = sled-agent/src/rack_setup/service.rs:477 retry_after = 197.850596ms sled_address = [fd00:1122:3344:101::1]:12345
7972024-08-23T21:52:37.195ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
7982024-08-23T21:52:37.195ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
7992024-08-23T21:52:37.195ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8002024-08-23T21:52:37.214ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
8012024-08-23T21:52:37.214ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8022024-08-23T21:52:37.233ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
8032024-08-23T21:52:37.233ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
8042024-08-23T21:52:37.250ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
8052024-08-23T21:52:37.250ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8062024-08-23T21:52:37.269ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
8072024-08-23T21:52:37.269ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8082024-08-23T21:52:37.287ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
8092024-08-23T21:52:37.287ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8102024-08-23T21:52:37.305ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
8112024-08-23T21:52:37.305ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8122024-08-23T21:52:37.324ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
8132024-08-23T21:52:37.324ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8142024-08-23T21:52:37.343ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
8152024-08-23T21:52:37.343ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
8162024-08-23T21:52:37.343ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
8172024-08-23T21:52:37.343ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
8182024-08-23T21:52:37.375ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
8192024-08-23T21:52:37.375ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
8202024-08-23T21:52:37.375ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
8212024-08-23T21:52:37.375ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
8222024-08-23T21:52:37.375ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
8232024-08-23T21:52:37.375ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657", TimeNotSynchronized), ("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb", TimeNotSynchronized), ("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8", TimeNotSynchronized), ("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd", TimeNotSynchronized), ("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 180717 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49912 req_id = b08cb024-880d-4f76-bd10-d079794a1295 response_code = 500 uri = /omicron-zones
8242024-08-23T21:52:37.376ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "b08cb024-880d-4f76-bd10-d079794a1295", "content-length": "124", "date": "Fri, 23 Aug 2024 21:52:36 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "b08cb024-880d-4f76-bd10-d079794a1295" } file = sled-agent/src/rack_setup/service.rs:477 retry_after = 327.839309ms sled_address = [fd00:1122:3344:101::1]:12345
8252024-08-23T21:52:37.705ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
8262024-08-23T21:52:37.705ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
8272024-08-23T21:52:37.705ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8282024-08-23T21:52:37.725ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
8292024-08-23T21:52:37.725ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8302024-08-23T21:52:37.744ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
8312024-08-23T21:52:37.744ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
8322024-08-23T21:52:37.762ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
8332024-08-23T21:52:37.762ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8342024-08-23T21:52:37.781ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
8352024-08-23T21:52:37.781ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8362024-08-23T21:52:37.799ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
8372024-08-23T21:52:37.799ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8382024-08-23T21:52:37.818ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
8392024-08-23T21:52:37.818ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8402024-08-23T21:52:37.837ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
8412024-08-23T21:52:37.837ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8422024-08-23T21:52:37.855ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
8432024-08-23T21:52:37.855ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
8442024-08-23T21:52:37.856ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
8452024-08-23T21:52:37.856ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
8462024-08-23T21:52:37.888ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
8472024-08-23T21:52:37.888ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
8482024-08-23T21:52:37.888ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
8492024-08-23T21:52:37.888ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
8502024-08-23T21:52:37.888ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
8512024-08-23T21:52:37.888ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd", TimeNotSynchronized), ("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8", TimeNotSynchronized), ("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb", TimeNotSynchronized), ("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657", TimeNotSynchronized), ("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 183467 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49912 req_id = 098dace5-b0fc-48ca-8f26-a8cc48721a2c response_code = 500 uri = /omicron-zones
8522024-08-23T21:52:37.888ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "098dace5-b0fc-48ca-8f26-a8cc48721a2c", "content-length": "124", "date": "Fri, 23 Aug 2024 21:52:37 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "098dace5-b0fc-48ca-8f26-a8cc48721a2c" } file = sled-agent/src/rack_setup/service.rs:477 retry_after = 232.83139ms sled_address = [fd00:1122:3344:101::1]:12345
8532024-08-23T21:52:38.123ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
8542024-08-23T21:52:38.123ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
8552024-08-23T21:52:38.123ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8562024-08-23T21:52:38.143ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
8572024-08-23T21:52:38.143ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8582024-08-23T21:52:38.160ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
8592024-08-23T21:52:38.160ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
8602024-08-23T21:52:38.177ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
8612024-08-23T21:52:38.177ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8622024-08-23T21:52:38.195ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
8632024-08-23T21:52:38.195ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8642024-08-23T21:52:38.212ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
8652024-08-23T21:52:38.212ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8662024-08-23T21:52:38.230ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
8672024-08-23T21:52:38.230ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8682024-08-23T21:52:38.249ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
8692024-08-23T21:52:38.249ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8702024-08-23T21:52:38.268ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
8712024-08-23T21:52:38.268ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
8722024-08-23T21:52:38.268ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
8732024-08-23T21:52:38.268ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
8742024-08-23T21:52:38.299ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
8752024-08-23T21:52:38.299ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
8762024-08-23T21:52:38.299ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
8772024-08-23T21:52:38.299ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
8782024-08-23T21:52:38.299ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
8792024-08-23T21:52:38.299ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb", TimeNotSynchronized), ("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd", TimeNotSynchronized), ("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855", TimeNotSynchronized), ("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657", TimeNotSynchronized), ("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 175222 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49912 req_id = be1db539-4b81-42d5-b1b5-d202c5d3a2f2 response_code = 500 uri = /omicron-zones
8802024-08-23T21:52:38.299ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "be1db539-4b81-42d5-b1b5-d202c5d3a2f2", "content-length": "124", "date": "Fri, 23 Aug 2024 21:52:37 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "be1db539-4b81-42d5-b1b5-d202c5d3a2f2" } file = sled-agent/src/rack_setup/service.rs:477 retry_after = 260.327947ms sled_address = [fd00:1122:3344:101::1]:12345
8812024-08-23T21:52:38.561ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
8822024-08-23T21:52:38.562ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
8832024-08-23T21:52:38.562ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8842024-08-23T21:52:38.582ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
8852024-08-23T21:52:38.582ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8862024-08-23T21:52:38.600ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
8872024-08-23T21:52:38.600ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
8882024-08-23T21:52:38.618ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
8892024-08-23T21:52:38.618ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8902024-08-23T21:52:38.636ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
8912024-08-23T21:52:38.636ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8922024-08-23T21:52:38.655ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
8932024-08-23T21:52:38.655ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8942024-08-23T21:52:38.673ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
8952024-08-23T21:52:38.673ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8962024-08-23T21:52:38.691ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
8972024-08-23T21:52:38.691ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8982024-08-23T21:52:38.709ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
8992024-08-23T21:52:38.709ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
9002024-08-23T21:52:38.709ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
9012024-08-23T21:52:38.709ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
9022024-08-23T21:52:38.742ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
9032024-08-23T21:52:38.742ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
9042024-08-23T21:52:38.742ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
9052024-08-23T21:52:38.742ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
9062024-08-23T21:52:38.742ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
9072024-08-23T21:52:38.742ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855", TimeNotSynchronized), ("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657", TimeNotSynchronized), ("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8", TimeNotSynchronized), ("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb", TimeNotSynchronized), ("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 179785 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49912 req_id = e9cdb75c-5ad0-4623-bec2-d85790da21c9 response_code = 500 uri = /omicron-zones
9082024-08-23T21:52:38.742ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "e9cdb75c-5ad0-4623-bec2-d85790da21c9", "content-length": "124", "date": "Fri, 23 Aug 2024 21:52:38 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "e9cdb75c-5ad0-4623-bec2-d85790da21c9" } file = sled-agent/src/rack_setup/service.rs:477 retry_after = 468.105964ms sled_address = [fd00:1122:3344:101::1]:12345
9092024-08-23T21:52:39.212ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
9102024-08-23T21:52:39.212ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
9112024-08-23T21:52:39.212ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9122024-08-23T21:52:39.232ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
9132024-08-23T21:52:39.232ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9142024-08-23T21:52:39.249ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
9152024-08-23T21:52:39.249ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
9162024-08-23T21:52:39.266ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
9172024-08-23T21:52:39.266ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9182024-08-23T21:52:39.285ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
9192024-08-23T21:52:39.285ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9202024-08-23T21:52:39.301ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
9212024-08-23T21:52:39.301ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9222024-08-23T21:52:39.317ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
9232024-08-23T21:52:39.317ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9242024-08-23T21:52:39.335ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
9252024-08-23T21:52:39.335ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9262024-08-23T21:52:39.354ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
9272024-08-23T21:52:39.354ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
9282024-08-23T21:52:39.354ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
9292024-08-23T21:52:39.354ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
9302024-08-23T21:52:39.387ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
9312024-08-23T21:52:39.387ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
9322024-08-23T21:52:39.387ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
9332024-08-23T21:52:39.387ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
9342024-08-23T21:52:39.387ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
9352024-08-23T21:52:39.387ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8", TimeNotSynchronized), ("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb", TimeNotSynchronized), ("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855", TimeNotSynchronized), ("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd", TimeNotSynchronized), ("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 175183 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49912 req_id = 3ea2b9fc-5259-4977-879c-21456a4a5a28 response_code = 500 uri = /omicron-zones
9362024-08-23T21:52:39.388ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "3ea2b9fc-5259-4977-879c-21456a4a5a28", "content-length": "124", "date": "Fri, 23 Aug 2024 21:52:39 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "3ea2b9fc-5259-4977-879c-21456a4a5a28" } file = sled-agent/src/rack_setup/service.rs:477 retry_after = 454.172644ms sled_address = [fd00:1122:3344:101::1]:12345
9372024-08-23T21:52:39.843ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
9382024-08-23T21:52:39.844ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
9392024-08-23T21:52:39.844ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9402024-08-23T21:52:39.864ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
9412024-08-23T21:52:39.864ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9422024-08-23T21:52:39.883ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
9432024-08-23T21:52:39.883ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
9442024-08-23T21:52:39.901ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
9452024-08-23T21:52:39.901ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9462024-08-23T21:52:39.919ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
9472024-08-23T21:52:39.919ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9482024-08-23T21:52:39.935ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
9492024-08-23T21:52:39.935ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9502024-08-23T21:52:39.953ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
9512024-08-23T21:52:39.953ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9522024-08-23T21:52:39.971ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
9532024-08-23T21:52:39.971ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9542024-08-23T21:52:39.988ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
9552024-08-23T21:52:39.989ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
9562024-08-23T21:52:39.989ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
9572024-08-23T21:52:39.989ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
9582024-08-23T21:52:40.020ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
9592024-08-23T21:52:40.020ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
9602024-08-23T21:52:40.020ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
9612024-08-23T21:52:40.021ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
9622024-08-23T21:52:40.021ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
9632024-08-23T21:52:40.021ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855", TimeNotSynchronized), ("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657", TimeNotSynchronized), ("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd", TimeNotSynchronized), ("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8", TimeNotSynchronized), ("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 176549 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49912 req_id = cb2750a8-5b34-4caa-aace-9a428d34ddba response_code = 500 uri = /omicron-zones
9642024-08-23T21:52:40.021ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "cb2750a8-5b34-4caa-aace-9a428d34ddba", "content-length": "124", "date": "Fri, 23 Aug 2024 21:52:39 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "cb2750a8-5b34-4caa-aace-9a428d34ddba" } file = sled-agent/src/rack_setup/service.rs:477 retry_after = 367.997458ms sled_address = [fd00:1122:3344:101::1]:12345
9652024-08-23T21:52:40.391ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
9662024-08-23T21:52:40.392ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
9672024-08-23T21:52:40.392ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9682024-08-23T21:52:40.412ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
9692024-08-23T21:52:40.412ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9702024-08-23T21:52:40.430ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
9712024-08-23T21:52:40.430ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
9722024-08-23T21:52:40.448ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
9732024-08-23T21:52:40.448ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9742024-08-23T21:52:40.467ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
9752024-08-23T21:52:40.467ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9762024-08-23T21:52:40.486ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
9772024-08-23T21:52:40.487ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9782024-08-23T21:52:40.504ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
9792024-08-23T21:52:40.505ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9802024-08-23T21:52:40.523ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
9812024-08-23T21:52:40.523ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9822024-08-23T21:52:40.541ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
9832024-08-23T21:52:40.541ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
9842024-08-23T21:52:40.541ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
9852024-08-23T21:52:40.541ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
9862024-08-23T21:52:40.572ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
9872024-08-23T21:52:40.572ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
9882024-08-23T21:52:40.572ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
9892024-08-23T21:52:40.572ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
9902024-08-23T21:52:40.572ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
9912024-08-23T21:52:40.572ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb", TimeNotSynchronized), ("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855", TimeNotSynchronized), ("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8", TimeNotSynchronized), ("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657", TimeNotSynchronized), ("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 180176 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49912 req_id = 400fc871-a220-4814-b3ae-a4181b0d9eed response_code = 500 uri = /omicron-zones
9922024-08-23T21:52:40.572ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "400fc871-a220-4814-b3ae-a4181b0d9eed", "content-length": "124", "date": "Fri, 23 Aug 2024 21:52:39 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "400fc871-a220-4814-b3ae-a4181b0d9eed" } file = sled-agent/src/rack_setup/service.rs:477 retry_after = 772.823381ms sled_address = [fd00:1122:3344:101::1]:12345
9932024-08-23T21:52:41.348ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
9942024-08-23T21:52:41.348ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
9952024-08-23T21:52:41.348ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9962024-08-23T21:52:41.369ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
9972024-08-23T21:52:41.369ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9982024-08-23T21:52:41.389ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
9992024-08-23T21:52:41.389ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
10002024-08-23T21:52:41.408ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
10012024-08-23T21:52:41.408ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10022024-08-23T21:52:41.428ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
10032024-08-23T21:52:41.428ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10042024-08-23T21:52:41.446ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
10052024-08-23T21:52:41.446ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10062024-08-23T21:52:41.464ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
10072024-08-23T21:52:41.464ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10082024-08-23T21:52:41.482ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
10092024-08-23T21:52:41.482ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10102024-08-23T21:52:41.501ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
10112024-08-23T21:52:41.501ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
10122024-08-23T21:52:41.501ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
10132024-08-23T21:52:41.501ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
10142024-08-23T21:52:41.532ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
10152024-08-23T21:52:41.532ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
10162024-08-23T21:52:41.532ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
10172024-08-23T21:52:41.532ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
10182024-08-23T21:52:41.532ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
10192024-08-23T21:52:41.532ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb", TimeNotSynchronized), ("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8", TimeNotSynchronized), ("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd", TimeNotSynchronized), ("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657", TimeNotSynchronized), ("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 183488 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49912 req_id = d021995e-f5d0-41cd-ba8b-e800418691d8 response_code = 500 uri = /omicron-zones
10202024-08-23T21:52:41.532ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "d021995e-f5d0-41cd-ba8b-e800418691d8", "content-length": "124", "date": "Fri, 23 Aug 2024 21:52:41 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "d021995e-f5d0-41cd-ba8b-e800418691d8" } file = sled-agent/src/rack_setup/service.rs:477 retry_after = 1.140448288s sled_address = [fd00:1122:3344:101::1]:12345
10212024-08-23T21:52:42.677ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
10222024-08-23T21:52:42.677ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
10232024-08-23T21:52:42.677ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10242024-08-23T21:52:42.698ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
10252024-08-23T21:52:42.698ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10262024-08-23T21:52:42.717ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
10272024-08-23T21:52:42.717ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
10282024-08-23T21:52:42.736ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
10292024-08-23T21:52:42.736ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10302024-08-23T21:52:42.755ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
10312024-08-23T21:52:42.755ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10322024-08-23T21:52:42.774ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
10332024-08-23T21:52:42.774ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10342024-08-23T21:52:42.792ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
10352024-08-23T21:52:42.792ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10362024-08-23T21:52:42.810ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
10372024-08-23T21:52:42.810ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10382024-08-23T21:52:42.828ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
10392024-08-23T21:52:42.829ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
10402024-08-23T21:52:42.829ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
10412024-08-23T21:52:42.829ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
10422024-08-23T21:52:42.861ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
10432024-08-23T21:52:42.861ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
10442024-08-23T21:52:42.861ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
10452024-08-23T21:52:42.861ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
10462024-08-23T21:52:42.861ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
10472024-08-23T21:52:42.861ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd", TimeNotSynchronized), ("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb", TimeNotSynchronized), ("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855", TimeNotSynchronized), ("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657", TimeNotSynchronized), ("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 183582 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49912 req_id = b1ca3424-6564-4fd8-80a5-bab9b27401c5 response_code = 500 uri = /omicron-zones
10482024-08-23T21:52:42.861ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "b1ca3424-6564-4fd8-80a5-bab9b27401c5", "content-length": "124", "date": "Fri, 23 Aug 2024 21:52:42 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "b1ca3424-6564-4fd8-80a5-bab9b27401c5" } file = sled-agent/src/rack_setup/service.rs:477 retry_after = 1.22132017s sled_address = [fd00:1122:3344:101::1]:12345
10492024-08-23T21:52:44.087ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
10502024-08-23T21:52:44.088ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
10512024-08-23T21:52:44.088ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10522024-08-23T21:52:44.108ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
10532024-08-23T21:52:44.108ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10542024-08-23T21:52:44.127ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
10552024-08-23T21:52:44.127ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
10562024-08-23T21:52:44.146ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
10572024-08-23T21:52:44.146ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10582024-08-23T21:52:44.166ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
10592024-08-23T21:52:44.166ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10602024-08-23T21:52:44.184ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
10612024-08-23T21:52:44.185ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10622024-08-23T21:52:44.203ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
10632024-08-23T21:52:44.203ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10642024-08-23T21:52:44.221ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
10652024-08-23T21:52:44.221ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10662024-08-23T21:52:44.240ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
10672024-08-23T21:52:44.240ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
10682024-08-23T21:52:44.240ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
10692024-08-23T21:52:44.240ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
10702024-08-23T21:52:44.272ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
10712024-08-23T21:52:44.272ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
10722024-08-23T21:52:44.272ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
10732024-08-23T21:52:44.272ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
10742024-08-23T21:52:44.272ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
10752024-08-23T21:52:44.273ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb", TimeNotSynchronized), ("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8", TimeNotSynchronized), ("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855", TimeNotSynchronized), ("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd", TimeNotSynchronized), ("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 184655 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49912 req_id = 41698440-58c0-49bd-875d-c9ccc53161e7 response_code = 500 uri = /omicron-zones
10762024-08-23T21:52:44.273ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "41698440-58c0-49bd-875d-c9ccc53161e7", "content-length": "124", "date": "Fri, 23 Aug 2024 21:52:44 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "41698440-58c0-49bd-875d-c9ccc53161e7" } file = sled-agent/src/rack_setup/service.rs:477 retry_after = 1.363105965s sled_address = [fd00:1122:3344:101::1]:12345
10772024-08-23T21:52:45.641ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
10782024-08-23T21:52:45.642ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
10792024-08-23T21:52:45.642ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10802024-08-23T21:52:45.662ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
10812024-08-23T21:52:45.662ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10822024-08-23T21:52:45.681ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
10832024-08-23T21:52:45.681ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
10842024-08-23T21:52:45.699ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
10852024-08-23T21:52:45.699ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10862024-08-23T21:52:45.716ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
10872024-08-23T21:52:45.716ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10882024-08-23T21:52:45.733ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
10892024-08-23T21:52:45.733ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10902024-08-23T21:52:45.752ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
10912024-08-23T21:52:45.752ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10922024-08-23T21:52:45.769ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
10932024-08-23T21:52:45.769ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10942024-08-23T21:52:45.786ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
10952024-08-23T21:52:45.786ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
10962024-08-23T21:52:45.787ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
10972024-08-23T21:52:45.787ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
10982024-08-23T21:52:45.818ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
10992024-08-23T21:52:45.818ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
11002024-08-23T21:52:45.818ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
11012024-08-23T21:52:45.818ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
11022024-08-23T21:52:45.818ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
11032024-08-23T21:52:45.818ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb", TimeNotSynchronized), ("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8", TimeNotSynchronized), ("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657", TimeNotSynchronized), ("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd", TimeNotSynchronized), ("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 176611 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49912 req_id = 0d6c0cf5-59cc-4203-ab46-f59eef1a4754 response_code = 500 uri = /omicron-zones
11042024-08-23T21:52:45.819ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "0d6c0cf5-59cc-4203-ab46-f59eef1a4754", "content-length": "124", "date": "Fri, 23 Aug 2024 21:52:45 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "0d6c0cf5-59cc-4203-ab46-f59eef1a4754" } file = sled-agent/src/rack_setup/service.rs:477 retry_after = 1.13880456s sled_address = [fd00:1122:3344:101::1]:12345
11052024-08-23T21:52:46.961ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
11062024-08-23T21:52:46.961ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
11072024-08-23T21:52:46.961ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11082024-08-23T21:52:46.982ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
11092024-08-23T21:52:46.982ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11102024-08-23T21:52:47.001ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
11112024-08-23T21:52:47.001ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
11122024-08-23T21:52:47.022ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
11132024-08-23T21:52:47.022ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11142024-08-23T21:52:47.041ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
11152024-08-23T21:52:47.041ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11162024-08-23T21:52:47.060ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
11172024-08-23T21:52:47.060ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11182024-08-23T21:52:47.078ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
11192024-08-23T21:52:47.078ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11202024-08-23T21:52:47.096ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
11212024-08-23T21:52:47.096ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11222024-08-23T21:52:47.114ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
11232024-08-23T21:52:47.114ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
11242024-08-23T21:52:47.114ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
11252024-08-23T21:52:47.114ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
11262024-08-23T21:52:47.147ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
11272024-08-23T21:52:47.147ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
11282024-08-23T21:52:47.147ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
11292024-08-23T21:52:47.147ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
11302024-08-23T21:52:47.147ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
11312024-08-23T21:52:47.147ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855", TimeNotSynchronized), ("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd", TimeNotSynchronized), ("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657", TimeNotSynchronized), ("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb", TimeNotSynchronized), ("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 185298 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49912 req_id = 3e2faf79-75b1-492b-8228-9b6afa614507 response_code = 500 uri = /omicron-zones
11322024-08-23T21:52:47.147ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "3e2faf79-75b1-492b-8228-9b6afa614507", "content-length": "124", "date": "Fri, 23 Aug 2024 21:52:47 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "3e2faf79-75b1-492b-8228-9b6afa614507" } file = sled-agent/src/rack_setup/service.rs:477 retry_after = 1.673425029s sled_address = [fd00:1122:3344:101::1]:12345
11332024-08-23T21:52:48.826ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
11342024-08-23T21:52:48.826ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
11352024-08-23T21:52:48.826ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11362024-08-23T21:52:48.847ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
11372024-08-23T21:52:48.847ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11382024-08-23T21:52:48.866ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
11392024-08-23T21:52:48.866ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
11402024-08-23T21:52:48.885ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
11412024-08-23T21:52:48.885ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11422024-08-23T21:52:48.904ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
11432024-08-23T21:52:48.904ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11442024-08-23T21:52:48.923ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
11452024-08-23T21:52:48.923ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11462024-08-23T21:52:48.941ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
11472024-08-23T21:52:48.941ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11482024-08-23T21:52:48.960ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
11492024-08-23T21:52:48.960ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11502024-08-23T21:52:48.979ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
11512024-08-23T21:52:48.979ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
11522024-08-23T21:52:48.979ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
11532024-08-23T21:52:48.979ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
11542024-08-23T21:52:49.013ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
11552024-08-23T21:52:49.013ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
11562024-08-23T21:52:49.013ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
11572024-08-23T21:52:49.013ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
11582024-08-23T21:52:49.013ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
11592024-08-23T21:52:49.013ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855", TimeNotSynchronized), ("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8", TimeNotSynchronized), ("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb", TimeNotSynchronized), ("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd", TimeNotSynchronized), ("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 186492 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49912 req_id = 40c8589a-cf01-4017-abdd-e1db14a621d0 response_code = 500 uri = /omicron-zones
11602024-08-23T21:52:49.013ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "40c8589a-cf01-4017-abdd-e1db14a621d0", "content-length": "124", "date": "Fri, 23 Aug 2024 21:52:48 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "40c8589a-cf01-4017-abdd-e1db14a621d0" } file = sled-agent/src/rack_setup/service.rs:477 retry_after = 2.686894919s sled_address = [fd00:1122:3344:101::1]:12345
11612024-08-23T21:52:49.106ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:46498
11622024-08-23T21:52:49.106ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 59 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:46498 req_id = af3b3144-aa80-46cc-9c46-32a20461c500 response_code = 200 uri = /baseboard
11632024-08-23T21:52:51.411ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 128.778684ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
11642024-08-23T21:52:51.709ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
11652024-08-23T21:52:51.709ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
11662024-08-23T21:52:51.709ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11672024-08-23T21:52:51.730ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
11682024-08-23T21:52:51.730ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11692024-08-23T21:52:51.750ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
11702024-08-23T21:52:51.750ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
11712024-08-23T21:52:51.768ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
11722024-08-23T21:52:51.768ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11732024-08-23T21:52:51.787ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
11742024-08-23T21:52:51.787ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11752024-08-23T21:52:51.806ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
11762024-08-23T21:52:51.806ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11772024-08-23T21:52:51.828ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
11782024-08-23T21:52:51.828ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11792024-08-23T21:52:51.846ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
11802024-08-23T21:52:51.846ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11812024-08-23T21:52:51.864ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
11822024-08-23T21:52:51.864ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
11832024-08-23T21:52:51.864ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
11842024-08-23T21:52:51.864ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
11852024-08-23T21:52:51.896ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
11862024-08-23T21:52:51.896ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
11872024-08-23T21:52:51.896ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
11882024-08-23T21:52:51.896ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
11892024-08-23T21:52:51.896ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
11902024-08-23T21:52:51.896ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657", TimeNotSynchronized), ("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855", TimeNotSynchronized), ("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd", TimeNotSynchronized), ("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8", TimeNotSynchronized), ("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 187050 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49912 req_id = 6b95d01c-0959-462f-905e-5472a0179ae4 response_code = 500 uri = /omicron-zones
11912024-08-23T21:52:51.896ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "6b95d01c-0959-462f-905e-5472a0179ae4", "content-length": "124", "date": "Fri, 23 Aug 2024 21:52:51 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "6b95d01c-0959-462f-905e-5472a0179ae4" } file = sled-agent/src/rack_setup/service.rs:477 retry_after = 1.578725118s sled_address = [fd00:1122:3344:101::1]:12345
11922024-08-23T21:52:53.481ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
11932024-08-23T21:52:53.482ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
11942024-08-23T21:52:53.482ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11952024-08-23T21:52:53.503ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
11962024-08-23T21:52:53.503ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11972024-08-23T21:52:53.522ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
11982024-08-23T21:52:53.522ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
11992024-08-23T21:52:53.542ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
12002024-08-23T21:52:53.542ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12012024-08-23T21:52:53.561ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
12022024-08-23T21:52:53.561ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12032024-08-23T21:52:53.580ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
12042024-08-23T21:52:53.580ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12052024-08-23T21:52:53.598ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
12062024-08-23T21:52:53.598ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12072024-08-23T21:52:53.616ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
12082024-08-23T21:52:53.616ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12092024-08-23T21:52:53.633ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
12102024-08-23T21:52:53.634ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
12112024-08-23T21:52:53.634ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
12122024-08-23T21:52:53.634ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
12132024-08-23T21:52:53.666ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
12142024-08-23T21:52:53.666ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
12152024-08-23T21:52:53.666ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
12162024-08-23T21:52:53.666ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
12172024-08-23T21:52:53.666ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
12182024-08-23T21:52:53.666ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd", TimeNotSynchronized), ("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657", TimeNotSynchronized), ("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb", TimeNotSynchronized), ("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8", TimeNotSynchronized), ("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 184011 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49912 req_id = fcd0a256-3931-4862-9407-d53330fc5d0f response_code = 500 uri = /omicron-zones
12192024-08-23T21:52:53.666ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "fcd0a256-3931-4862-9407-d53330fc5d0f", "content-length": "124", "date": "Fri, 23 Aug 2024 21:52:53 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "fcd0a256-3931-4862-9407-d53330fc5d0f" } file = sled-agent/src/rack_setup/service.rs:477 retry_after = 3.231456905s sled_address = [fd00:1122:3344:101::1]:12345
12202024-08-23T21:52:56.907ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
12212024-08-23T21:52:56.908ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
12222024-08-23T21:52:56.908ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12232024-08-23T21:52:56.928ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
12242024-08-23T21:52:56.928ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12252024-08-23T21:52:56.948ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
12262024-08-23T21:52:56.948ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
12272024-08-23T21:52:56.966ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
12282024-08-23T21:52:56.966ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12292024-08-23T21:52:56.984ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
12302024-08-23T21:52:56.984ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12312024-08-23T21:52:57.002ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
12322024-08-23T21:52:57.002ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12332024-08-23T21:52:57.020ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
12342024-08-23T21:52:57.020ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12352024-08-23T21:52:57.039ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
12362024-08-23T21:52:57.039ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12372024-08-23T21:52:57.056ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
12382024-08-23T21:52:57.056ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
12392024-08-23T21:52:57.056ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
12402024-08-23T21:52:57.057ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
12412024-08-23T21:52:57.089ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
12422024-08-23T21:52:57.089ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
12432024-08-23T21:52:57.089ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
12442024-08-23T21:52:57.089ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
12452024-08-23T21:52:57.089ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
12462024-08-23T21:52:57.089ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd", TimeNotSynchronized), ("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8", TimeNotSynchronized), ("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657", TimeNotSynchronized), ("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb", TimeNotSynchronized), ("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 181122 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49912 req_id = 72f608d6-a1cb-4fae-a54e-acf069e50970 response_code = 500 uri = /omicron-zones
12472024-08-23T21:52:57.089ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "72f608d6-a1cb-4fae-a54e-acf069e50970", "content-length": "124", "date": "Fri, 23 Aug 2024 21:52:56 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "72f608d6-a1cb-4fae-a54e-acf069e50970" } file = sled-agent/src/rack_setup/service.rs:477 retry_after = 3.116733437s sled_address = [fd00:1122:3344:101::1]:12345
12482024-08-23T21:53:00.215ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
12492024-08-23T21:53:00.216ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
12502024-08-23T21:53:00.216ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12512024-08-23T21:53:00.237ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
12522024-08-23T21:53:00.237ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12532024-08-23T21:53:00.256ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
12542024-08-23T21:53:00.256ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
12552024-08-23T21:53:00.275ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
12562024-08-23T21:53:00.275ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12572024-08-23T21:53:00.294ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
12582024-08-23T21:53:00.294ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12592024-08-23T21:53:00.313ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
12602024-08-23T21:53:00.313ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12612024-08-23T21:53:00.331ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
12622024-08-23T21:53:00.331ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12632024-08-23T21:53:00.349ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
12642024-08-23T21:53:00.349ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12652024-08-23T21:53:00.367ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
12662024-08-23T21:53:00.367ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
12672024-08-23T21:53:00.367ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
12682024-08-23T21:53:00.367ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
12692024-08-23T21:53:00.399ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
12702024-08-23T21:53:00.399ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
12712024-08-23T21:53:00.399ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
12722024-08-23T21:53:00.399ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
12732024-08-23T21:53:00.399ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
12742024-08-23T21:53:00.399ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb", TimeNotSynchronized), ("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8", TimeNotSynchronized), ("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd", TimeNotSynchronized), ("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657", TimeNotSynchronized), ("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 182824 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49912 req_id = 6ffb2182-e4ac-4007-a935-58fd89bf0838 response_code = 500 uri = /omicron-zones
12752024-08-23T21:53:00.399ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "6ffb2182-e4ac-4007-a935-58fd89bf0838", "content-length": "124", "date": "Fri, 23 Aug 2024 21:53:00 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "6ffb2182-e4ac-4007-a935-58fd89bf0838" } file = sled-agent/src/rack_setup/service.rs:477 retry_after = 3.046296379s sled_address = [fd00:1122:3344:101::1]:12345
12762024-08-23T21:53:03.455ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
12772024-08-23T21:53:03.456ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
12782024-08-23T21:53:03.456ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12792024-08-23T21:53:03.476ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
12802024-08-23T21:53:03.476ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12812024-08-23T21:53:03.493ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
12822024-08-23T21:53:03.493ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
12832024-08-23T21:53:03.508ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
12842024-08-23T21:53:03.508ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12852024-08-23T21:53:03.523ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
12862024-08-23T21:53:03.523ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12872024-08-23T21:53:03.541ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
12882024-08-23T21:53:03.541ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12892024-08-23T21:53:03.559ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
12902024-08-23T21:53:03.559ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12912024-08-23T21:53:03.577ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
12922024-08-23T21:53:03.577ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12932024-08-23T21:53:03.594ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
12942024-08-23T21:53:03.594ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
12952024-08-23T21:53:03.594ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
12962024-08-23T21:53:03.594ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
12972024-08-23T21:53:03.625ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
12982024-08-23T21:53:03.625ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
12992024-08-23T21:53:03.625ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
13002024-08-23T21:53:03.625ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
13012024-08-23T21:53:03.625ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
13022024-08-23T21:53:03.626ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657", TimeNotSynchronized), ("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb", TimeNotSynchronized), ("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8", TimeNotSynchronized), ("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd", TimeNotSynchronized), ("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 169636 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49912 req_id = 21b7ee46-6bd2-4d3f-b620-8b0e0dce8de9 response_code = 500 uri = /omicron-zones
13032024-08-23T21:53:03.626ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "21b7ee46-6bd2-4d3f-b620-8b0e0dce8de9", "content-length": "124", "date": "Fri, 23 Aug 2024 21:53:03 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "21b7ee46-6bd2-4d3f-b620-8b0e0dce8de9" } file = sled-agent/src/rack_setup/service.rs:477 retry_after = 3.989540301s sled_address = [fd00:1122:3344:101::1]:12345
13042024-08-23T21:53:06.615ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 444.168388ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
13052024-08-23T21:53:07.627ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
13062024-08-23T21:53:07.628ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
13072024-08-23T21:53:07.628ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13082024-08-23T21:53:07.649ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
13092024-08-23T21:53:07.649ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13102024-08-23T21:53:07.670ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
13112024-08-23T21:53:07.670ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
13122024-08-23T21:53:07.689ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
13132024-08-23T21:53:07.689ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13142024-08-23T21:53:07.708ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
13152024-08-23T21:53:07.708ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13162024-08-23T21:53:07.726ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
13172024-08-23T21:53:07.726ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13182024-08-23T21:53:07.745ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
13192024-08-23T21:53:07.745ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13202024-08-23T21:53:07.763ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
13212024-08-23T21:53:07.763ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13222024-08-23T21:53:07.780ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
13232024-08-23T21:53:07.780ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
13242024-08-23T21:53:07.781ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
13252024-08-23T21:53:07.781ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
13262024-08-23T21:53:07.814ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
13272024-08-23T21:53:07.814ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
13282024-08-23T21:53:07.814ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
13292024-08-23T21:53:07.814ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
13302024-08-23T21:53:07.814ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
13312024-08-23T21:53:07.814ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb", TimeNotSynchronized), ("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657", TimeNotSynchronized), ("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8", TimeNotSynchronized), ("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855", TimeNotSynchronized), ("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 186275 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49912 req_id = f8f13418-f7b0-4982-beba-49f0c6e5b9df response_code = 500 uri = /omicron-zones
13322024-08-23T21:53:07.814ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "f8f13418-f7b0-4982-beba-49f0c6e5b9df", "content-length": "124", "date": "Fri, 23 Aug 2024 21:53:07 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "f8f13418-f7b0-4982-beba-49f0c6e5b9df" } file = sled-agent/src/rack_setup/service.rs:477 retry_after = 3.569827878s sled_address = [fd00:1122:3344:101::1]:12345
13332024-08-23T21:53:11.395ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
13342024-08-23T21:53:11.395ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
13352024-08-23T21:53:11.395ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13362024-08-23T21:53:11.416ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
13372024-08-23T21:53:11.416ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13382024-08-23T21:53:11.436ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
13392024-08-23T21:53:11.436ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
13402024-08-23T21:53:11.455ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
13412024-08-23T21:53:11.455ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13422024-08-23T21:53:11.473ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
13432024-08-23T21:53:11.473ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13442024-08-23T21:53:11.492ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
13452024-08-23T21:53:11.492ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13462024-08-23T21:53:11.511ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
13472024-08-23T21:53:11.511ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13482024-08-23T21:53:11.530ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
13492024-08-23T21:53:11.530ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13502024-08-23T21:53:11.548ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
13512024-08-23T21:53:11.548ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
13522024-08-23T21:53:11.548ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
13532024-08-23T21:53:11.548ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
13542024-08-23T21:53:11.581ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
13552024-08-23T21:53:11.581ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
13562024-08-23T21:53:11.581ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
13572024-08-23T21:53:11.581ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
13582024-08-23T21:53:11.581ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
13592024-08-23T21:53:11.581ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8", TimeNotSynchronized), ("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657", TimeNotSynchronized), ("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855", TimeNotSynchronized), ("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb", TimeNotSynchronized), ("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 185298 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49912 req_id = ed4ed590-a0d9-4bb2-bfca-f10a930a9a7c response_code = 500 uri = /omicron-zones
13602024-08-23T21:53:11.581ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "ed4ed590-a0d9-4bb2-bfca-f10a930a9a7c", "content-length": "124", "date": "Fri, 23 Aug 2024 21:53:11 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "ed4ed590-a0d9-4bb2-bfca-f10a930a9a7c" } file = sled-agent/src/rack_setup/service.rs:477 retry_after = 3.998121158s sled_address = [fd00:1122:3344:101::1]:12345
13612024-08-23T21:53:15.592ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
13622024-08-23T21:53:15.592ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
13632024-08-23T21:53:15.592ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13642024-08-23T21:53:15.613ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
13652024-08-23T21:53:15.613ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13662024-08-23T21:53:15.633ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
13672024-08-23T21:53:15.633ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
13682024-08-23T21:53:15.652ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
13692024-08-23T21:53:15.652ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13702024-08-23T21:53:15.672ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
13712024-08-23T21:53:15.673ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13722024-08-23T21:53:15.693ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
13732024-08-23T21:53:15.693ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13742024-08-23T21:53:15.712ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
13752024-08-23T21:53:15.712ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13762024-08-23T21:53:15.731ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
13772024-08-23T21:53:15.731ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13782024-08-23T21:53:15.749ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
13792024-08-23T21:53:15.749ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
13802024-08-23T21:53:15.750ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
13812024-08-23T21:53:15.750ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
13822024-08-23T21:53:15.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
13832024-08-23T21:53:15.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
13842024-08-23T21:53:15.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
13852024-08-23T21:53:15.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
13862024-08-23T21:53:15.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
13872024-08-23T21:53:15.782ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657", TimeNotSynchronized), ("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855", TimeNotSynchronized), ("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8", TimeNotSynchronized), ("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd", TimeNotSynchronized), ("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 190116 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49912 req_id = ae14b2c4-9131-463a-b959-13d518b27fa0 response_code = 500 uri = /omicron-zones
13882024-08-23T21:53:15.782ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "ae14b2c4-9131-463a-b959-13d518b27fa0", "content-length": "124", "date": "Fri, 23 Aug 2024 21:53:15 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "ae14b2c4-9131-463a-b959-13d518b27fa0" } file = sled-agent/src/rack_setup/service.rs:477 retry_after = 10.694990668s sled_address = [fd00:1122:3344:101::1]:12345
13892024-08-23T21:53:19.223ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:40029
13902024-08-23T21:53:19.223ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 44 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:40029 req_id = 11bf1259-1df2-4997-a63c-ee9d0a28c942 response_code = 200 uri = /baseboard
13912024-08-23T21:53:22.137ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 991.885065ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
13922024-08-23T21:53:26.508ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
13932024-08-23T21:53:26.508ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
13942024-08-23T21:53:26.508ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13952024-08-23T21:53:26.530ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
13962024-08-23T21:53:26.530ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13972024-08-23T21:53:26.549ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
13982024-08-23T21:53:26.549ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
13992024-08-23T21:53:26.567ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
14002024-08-23T21:53:26.567ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14012024-08-23T21:53:26.587ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
14022024-08-23T21:53:26.587ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14032024-08-23T21:53:26.605ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
14042024-08-23T21:53:26.605ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14052024-08-23T21:53:26.624ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
14062024-08-23T21:53:26.624ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14072024-08-23T21:53:26.642ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
14082024-08-23T21:53:26.642ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14092024-08-23T21:53:26.660ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
14102024-08-23T21:53:26.660ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
14112024-08-23T21:53:26.661ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
14122024-08-23T21:53:26.661ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
14132024-08-23T21:53:26.692ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
14142024-08-23T21:53:26.692ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
14152024-08-23T21:53:26.692ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
14162024-08-23T21:53:26.692ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
14172024-08-23T21:53:26.692ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
14182024-08-23T21:53:26.693ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657", TimeNotSynchronized), ("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855", TimeNotSynchronized), ("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb", TimeNotSynchronized), ("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8", TimeNotSynchronized), ("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 184335 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49912 req_id = c3fbb6f8-9988-4f00-8c66-c40e60e7082a response_code = 500 uri = /omicron-zones
14192024-08-23T21:53:26.693ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "c3fbb6f8-9988-4f00-8c66-c40e60e7082a", "content-length": "124", "date": "Fri, 23 Aug 2024 21:53:26 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "c3fbb6f8-9988-4f00-8c66-c40e60e7082a" } file = sled-agent/src/rack_setup/service.rs:477 retry_after = 10.821984312s sled_address = [fd00:1122:3344:101::1]:12345
14202024-08-23T21:53:37.546ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
14212024-08-23T21:53:37.546ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
14222024-08-23T21:53:37.546ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14232024-08-23T21:53:37.569ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
14242024-08-23T21:53:37.569ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14252024-08-23T21:53:37.587ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
14262024-08-23T21:53:37.588ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
14272024-08-23T21:53:37.607ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
14282024-08-23T21:53:37.607ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14292024-08-23T21:53:37.626ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
14302024-08-23T21:53:37.626ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14312024-08-23T21:53:37.645ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
14322024-08-23T21:53:37.645ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14332024-08-23T21:53:37.663ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
14342024-08-23T21:53:37.663ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14352024-08-23T21:53:37.682ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
14362024-08-23T21:53:37.682ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14372024-08-23T21:53:37.701ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
14382024-08-23T21:53:37.701ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
14392024-08-23T21:53:37.701ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
14402024-08-23T21:53:37.701ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
14412024-08-23T21:53:37.734ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
14422024-08-23T21:53:37.734ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
14432024-08-23T21:53:37.734ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
14442024-08-23T21:53:37.734ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
14452024-08-23T21:53:37.734ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
14462024-08-23T21:53:37.734ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657", TimeNotSynchronized), ("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb", TimeNotSynchronized), ("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8", TimeNotSynchronized), ("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855", TimeNotSynchronized), ("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 187413 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49912 req_id = 44e66ef1-a1b3-486f-8561-97b1c876a8c7 response_code = 500 uri = /omicron-zones
14472024-08-23T21:53:37.734ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "44e66ef1-a1b3-486f-8561-97b1c876a8c7", "content-length": "124", "date": "Fri, 23 Aug 2024 21:53:37 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "44e66ef1-a1b3-486f-8561-97b1c876a8c7" } file = sled-agent/src/rack_setup/service.rs:477 retry_after = 18.268614743s sled_address = [fd00:1122:3344:101::1]:12345
14482024-08-23T21:53:38.207ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 1.789267315s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
14492024-08-23T21:53:49.337ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:47150
14502024-08-23T21:53:49.337ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 32 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:47150 req_id = 50cc4b52-0f8c-4a57-9111-8e5ad5c2b583 response_code = 200 uri = /baseboard
14512024-08-23T21:53:55.078ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 4.052496193s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
14522024-08-23T21:53:56.054ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
14532024-08-23T21:53:56.054ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
14542024-08-23T21:53:56.054ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14552024-08-23T21:53:56.076ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
14562024-08-23T21:53:56.076ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14572024-08-23T21:53:56.096ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
14582024-08-23T21:53:56.096ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
14592024-08-23T21:53:56.114ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
14602024-08-23T21:53:56.114ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14612024-08-23T21:53:56.134ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
14622024-08-23T21:53:56.134ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14632024-08-23T21:53:56.151ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
14642024-08-23T21:53:56.151ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14652024-08-23T21:53:56.171ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
14662024-08-23T21:53:56.171ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14672024-08-23T21:53:56.189ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
14682024-08-23T21:53:56.189ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14692024-08-23T21:53:56.208ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
14702024-08-23T21:53:56.208ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
14712024-08-23T21:53:56.208ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
14722024-08-23T21:53:56.208ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
14732024-08-23T21:53:56.241ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
14742024-08-23T21:53:56.241ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
14752024-08-23T21:53:56.241ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
14762024-08-23T21:53:56.241ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
14772024-08-23T21:53:56.241ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
14782024-08-23T21:53:56.241ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd", TimeNotSynchronized), ("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657", TimeNotSynchronized), ("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855", TimeNotSynchronized), ("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb", TimeNotSynchronized), ("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 186864 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49912 req_id = 527405cb-c938-47a6-836b-99c481eb9742 response_code = 500 uri = /omicron-zones
14792024-08-23T21:53:56.241ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "527405cb-c938-47a6-836b-99c481eb9742", "content-length": "124", "date": "Fri, 23 Aug 2024 21:53:56 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "527405cb-c938-47a6-836b-99c481eb9742" } file = sled-agent/src/rack_setup/service.rs:477 retry_after = 10.052432848s sled_address = [fd00:1122:3344:101::1]:12345
14802024-08-23T21:54:06.322ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
14812024-08-23T21:54:06.322ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
14822024-08-23T21:54:06.322ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14832024-08-23T21:54:06.344ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
14842024-08-23T21:54:06.344ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14852024-08-23T21:54:06.362ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
14862024-08-23T21:54:06.362ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
14872024-08-23T21:54:06.380ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
14882024-08-23T21:54:06.380ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14892024-08-23T21:54:06.401ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
14902024-08-23T21:54:06.401ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14912024-08-23T21:54:06.421ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
14922024-08-23T21:54:06.421ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14932024-08-23T21:54:06.439ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
14942024-08-23T21:54:06.439ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14952024-08-23T21:54:06.458ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
14962024-08-23T21:54:06.458ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14972024-08-23T21:54:06.477ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
14982024-08-23T21:54:06.477ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
14992024-08-23T21:54:06.477ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
15002024-08-23T21:54:06.477ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
15012024-08-23T21:54:06.510ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
15022024-08-23T21:54:06.510ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
15032024-08-23T21:54:06.510ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
15042024-08-23T21:54:06.510ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
15052024-08-23T21:54:06.510ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
15062024-08-23T21:54:06.510ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657", TimeNotSynchronized), ("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd", TimeNotSynchronized), ("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb", TimeNotSynchronized), ("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8", TimeNotSynchronized), ("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 187762 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49912 req_id = ecaf14cd-c5a2-4769-bc38-795ea93538bf response_code = 500 uri = /omicron-zones
15072024-08-23T21:54:06.510ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "ecaf14cd-c5a2-4769-bc38-795ea93538bf", "content-length": "124", "date": "Fri, 23 Aug 2024 21:54:06 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "ecaf14cd-c5a2-4769-bc38-795ea93538bf" } file = sled-agent/src/rack_setup/service.rs:477 retry_after = 16.323152492s sled_address = [fd00:1122:3344:101::1]:12345
15082024-08-23T21:54:14.221ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 11.17373946s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
15092024-08-23T21:54:19.451ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:36101
15102024-08-23T21:54:19.451ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 52 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:36101 req_id = 7bcd86b3-09a0-4d45-96f8-a43a574a281d response_code = 200 uri = /baseboard
15112024-08-23T21:54:22.879ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
15122024-08-23T21:54:22.879ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
15132024-08-23T21:54:22.879ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15142024-08-23T21:54:22.901ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
15152024-08-23T21:54:22.901ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15162024-08-23T21:54:22.921ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
15172024-08-23T21:54:22.921ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
15182024-08-23T21:54:22.940ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
15192024-08-23T21:54:22.940ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15202024-08-23T21:54:22.959ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
15212024-08-23T21:54:22.959ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15222024-08-23T21:54:22.977ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
15232024-08-23T21:54:22.977ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15242024-08-23T21:54:22.997ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
15252024-08-23T21:54:22.997ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15262024-08-23T21:54:23.016ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
15272024-08-23T21:54:23.016ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15282024-08-23T21:54:23.035ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
15292024-08-23T21:54:23.035ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
15302024-08-23T21:54:23.036ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
15312024-08-23T21:54:23.036ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
15322024-08-23T21:54:23.068ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
15332024-08-23T21:54:23.068ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
15342024-08-23T21:54:23.068ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
15352024-08-23T21:54:23.068ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
15362024-08-23T21:54:23.068ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
15372024-08-23T21:54:23.068ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657", TimeNotSynchronized), ("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8", TimeNotSynchronized), ("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd", TimeNotSynchronized), ("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855", TimeNotSynchronized), ("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 188450 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49912 req_id = 3d1a9170-1420-478b-a0ca-a870d2400aa8 response_code = 500 uri = /omicron-zones
15382024-08-23T21:54:23.068ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "3d1a9170-1420-478b-a0ca-a870d2400aa8", "content-length": "124", "date": "Fri, 23 Aug 2024 21:54:23 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "3d1a9170-1420-478b-a0ca-a870d2400aa8" } file = sled-agent/src/rack_setup/service.rs:477 retry_after = 25.630528577s sled_address = [fd00:1122:3344:101::1]:12345
15392024-08-23T21:54:40.502ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 19.482206901s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
15402024-08-23T21:54:48.770ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
15412024-08-23T21:54:48.770ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
15422024-08-23T21:54:48.770ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15432024-08-23T21:54:48.793ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
15442024-08-23T21:54:48.793ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15452024-08-23T21:54:48.813ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
15462024-08-23T21:54:48.813ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
15472024-08-23T21:54:48.832ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
15482024-08-23T21:54:48.832ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15492024-08-23T21:54:48.850ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
15502024-08-23T21:54:48.850ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15512024-08-23T21:54:48.870ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
15522024-08-23T21:54:48.870ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15532024-08-23T21:54:48.888ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
15542024-08-23T21:54:48.888ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15552024-08-23T21:54:48.906ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
15562024-08-23T21:54:48.906ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15572024-08-23T21:54:48.925ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
15582024-08-23T21:54:48.925ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
15592024-08-23T21:54:48.926ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
15602024-08-23T21:54:48.926ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
15612024-08-23T21:54:48.958ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
15622024-08-23T21:54:48.958ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
15632024-08-23T21:54:48.958ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
15642024-08-23T21:54:48.958ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
15652024-08-23T21:54:48.958ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
15662024-08-23T21:54:48.958ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd", TimeNotSynchronized), ("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855", TimeNotSynchronized), ("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657", TimeNotSynchronized), ("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8", TimeNotSynchronized), ("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 187450 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49912 req_id = 77f93f36-ee8c-44f9-aec9-54ba57e05d20 response_code = 500 uri = /omicron-zones
15672024-08-23T21:54:48.958ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "77f93f36-ee8c-44f9-aec9-54ba57e05d20", "content-length": "124", "date": "Fri, 23 Aug 2024 21:54:48 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "77f93f36-ee8c-44f9-aec9-54ba57e05d20" } file = sled-agent/src/rack_setup/service.rs:477 retry_after = 20.720843812s sled_address = [fd00:1122:3344:101::1]:12345
15682024-08-23T21:54:49.565ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:49223
15692024-08-23T21:54:49.565ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 40 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:49223 req_id = a6b3d9d5-6b98-4975-b1bc-9278de989517 response_code = 200 uri = /baseboard
15702024-08-23T21:55:09.737ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
15712024-08-23T21:55:09.737ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
15722024-08-23T21:55:09.737ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15732024-08-23T21:55:09.759ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
15742024-08-23T21:55:09.759ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15752024-08-23T21:55:09.779ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
15762024-08-23T21:55:09.779ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
15772024-08-23T21:55:09.798ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
15782024-08-23T21:55:09.798ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15792024-08-23T21:55:09.815ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
15802024-08-23T21:55:09.816ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15812024-08-23T21:55:09.834ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
15822024-08-23T21:55:09.834ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15832024-08-23T21:55:09.853ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
15842024-08-23T21:55:09.853ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15852024-08-23T21:55:09.872ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
15862024-08-23T21:55:09.872ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15872024-08-23T21:55:09.890ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
15882024-08-23T21:55:09.890ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
15892024-08-23T21:55:09.890ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
15902024-08-23T21:55:09.890ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
15912024-08-23T21:55:09.922ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
15922024-08-23T21:55:09.922ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
15932024-08-23T21:55:09.922ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
15942024-08-23T21:55:09.922ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
15952024-08-23T21:55:09.922ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3249 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
15962024-08-23T21:55:09.922ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8", TimeNotSynchronized), ("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd", TimeNotSynchronized), ("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855", TimeNotSynchronized), ("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657", TimeNotSynchronized), ("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 184954 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49912 req_id = 5e47ec52-beaf-4992-b6ca-e41b9e75a690 response_code = 500 uri = /omicron-zones
15972024-08-23T21:55:09.923ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "5e47ec52-beaf-4992-b6ca-e41b9e75a690", "content-length": "124", "date": "Fri, 23 Aug 2024 21:55:09 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "5e47ec52-beaf-4992-b6ca-e41b9e75a690" } file = sled-agent/src/rack_setup/service.rs:477 retry_after = 51.161663737s sled_address = [fd00:1122:3344:101::1]:12345
15982024-08-23T21:55:15.115ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 33.020207597s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
15992024-08-23T21:55:19.680ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:57000
16002024-08-23T21:55:19.680ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 30 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:57000 req_id = 29f57eda-6e11-43e3-ada7-974ffb67f0bf response_code = 200 uri = /baseboard
16012024-08-23T21:55:49.754ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:56673
16022024-08-23T21:55:49.754ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 33 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:56673 req_id = 369556e4-9102-4424-90a7-226f17b4b552 response_code = 200 uri = /baseboard
16032024-08-23T21:56:01.153ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
16042024-08-23T21:56:01.154ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
16052024-08-23T21:56:01.154ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
16062024-08-23T21:56:01.177ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
16072024-08-23T21:56:01.177ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
16082024-08-23T21:56:01.196ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
16092024-08-23T21:56:01.196ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
16102024-08-23T21:56:01.213ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
16112024-08-23T21:56:01.213ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16122024-08-23T21:56:01.231ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
16132024-08-23T21:56:01.231ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16142024-08-23T21:56:01.250ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
16152024-08-23T21:56:01.250ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16162024-08-23T21:56:01.268ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
16172024-08-23T21:56:01.268ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16182024-08-23T21:56:01.287ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
16192024-08-23T21:56:01.287ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16202024-08-23T21:56:01.306ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
16212024-08-23T21:56:01.306ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
16222024-08-23T21:56:01.307ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
16232024-08-23T21:56:01.307ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
16242024-08-23T21:56:01.393ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
16252024-08-23T21:56:01.393ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
16262024-08-23T21:56:01.393ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
16272024-08-23T21:56:01.393ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
16282024-08-23T21:56:01.393ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
16292024-08-23T21:56:01.488ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657 file = illumos-utils/src/zone.rs:316
16302024-08-23T21:56:01.492ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb file = illumos-utils/src/zone.rs:316
16312024-08-23T21:56:01.496ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855 file = illumos-utils/src/zone.rs:316
16322024-08-23T21:56:01.501ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8 file = illumos-utils/src/zone.rs:316
16332024-08-23T21:56:01.505ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd file = illumos-utils/src/zone.rs:316
16342024-08-23T21:56:01.510ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657 file = illumos-utils/src/zone.rs:354
16352024-08-23T21:56:01.515ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb file = illumos-utils/src/zone.rs:354
16362024-08-23T21:56:01.519ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855 file = illumos-utils/src/zone.rs:354
16372024-08-23T21:56:01.523ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8 file = illumos-utils/src/zone.rs:354
16382024-08-23T21:56:01.528ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd file = illumos-utils/src/zone.rs:354
16392024-08-23T21:56:03.227ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 95.85143174s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
16402024-08-23T21:56:04.450ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService7'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::7]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
16412024-08-23T21:56:04.450ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
16422024-08-23T21:56:04.481ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService6'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::4'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='32a152fc-12d5-4442-9ec7-44167f70e8eb'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::4]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
16432024-08-23T21:56:04.481ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
16442024-08-23T21:56:04.554ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService5'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::3'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::3]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
16452024-08-23T21:56:04.554ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
16462024-08-23T21:56:04.604ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855") failed: Property not found. retry in 47.944608ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
16472024-08-23T21:56:04.625ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb") failed: Property not found. retry in 41.041384ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
16482024-08-23T21:56:04.626ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService8'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='a32a4cfc-b6ee-416e-8738-c10a4146b4f8'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::5]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
16492024-08-23T21:56:04.626ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
16502024-08-23T21:56:04.632ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService9'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='5749b096-fa59-4034-bf5c-f4e555b65bfd'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::6]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
16512024-08-23T21:56:04.632ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
16522024-08-23T21:56:04.662ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855") failed: Property not found. retry in 94.72605ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
16532024-08-23T21:56:04.695ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb") failed: Property not found. retry in 68.891223ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
16542024-08-23T21:56:04.900ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855") failed: Property not found. retry in 208.579833ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
16552024-08-23T21:56:04.967ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657") failed: Property not found. retry in 46.248382ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
16562024-08-23T21:56:04.990ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb") failed: Property not found. retry in 289.950401ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
16572024-08-23T21:56:05.077ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657") failed: Property not found. retry in 97.290575ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
16582024-08-23T21:56:05.181ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855") failed: Property not found. retry in 368.550958ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
16592024-08-23T21:56:05.202ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657") failed: Property not found. retry in 254.816185ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
16602024-08-23T21:56:05.232ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8") failed: Property not found. retry in 66.803349ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
16612024-08-23T21:56:05.701ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd") failed: Property not found. retry in 70.099362ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
16622024-08-23T21:56:05.744ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657") failed: Property not found. retry in 466.179813ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
16632024-08-23T21:56:05.975ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb") failed: Property not found. retry in 252.450263ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
16642024-08-23T21:56:06.067ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855") failed: Property not found. retry in 703.485401ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
16652024-08-23T21:56:06.150ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8") failed: Property not found. retry in 137.942421ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
16662024-08-23T21:56:06.201ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd") failed: Property not found. retry in 135.007436ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
16672024-08-23T21:56:06.300ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657") failed: Property not found. retry in 1.12953637s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
16682024-08-23T21:56:06.349ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb") failed: Property not found. retry in 1.099106132s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
16692024-08-23T21:56:06.390ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8") failed: Property not found. retry in 141.785154ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
16702024-08-23T21:56:06.475ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd") failed: Property not found. retry in 215.933822ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
16712024-08-23T21:56:06.672ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8") failed: Property not found. retry in 282.277102ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
16722024-08-23T21:56:06.741ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd") failed: Property not found. retry in 574.820327ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
16732024-08-23T21:56:06.836ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855") failed: Property not found. retry in 512.572868ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
16742024-08-23T21:56:06.984ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8") failed: Property not found. retry in 535.376473ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
16752024-08-23T21:56:07.327ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd") failed: Property not found. retry in 1.131177717s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
16762024-08-23T21:56:07.359ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855") failed: Property not found. retry in 1.356130097s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
16772024-08-23T21:56:07.441ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657") failed: Property not found. retry in 1.030984555s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
16782024-08-23T21:56:07.460ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb") failed: Property not found. retry in 1.132097229s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
16792024-08-23T21:56:07.529ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8") failed: Property not found. retry in 1.328973996s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
16802024-08-23T21:56:08.468ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd") failed: Property not found. retry in 634.03747ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
16812024-08-23T21:56:08.484ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657") failed: Property not found. retry in 1.488064218s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
16822024-08-23T21:56:08.601ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb") failed: Property not found. retry in 1.262911269s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
16832024-08-23T21:56:08.725ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855") failed: Property not found. retry in 979.240081ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
16842024-08-23T21:56:08.869ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8") failed: Property not found. retry in 808.205963ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
16852024-08-23T21:56:09.114ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd") failed: Property not found. retry in 663.341246ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
16862024-08-23T21:56:09.687ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8") failed: Property not found. retry in 860.085479ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
16872024-08-23T21:56:09.714ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855") failed: Property not found. retry in 688.048908ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
16882024-08-23T21:56:09.786ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd") failed: Property not found. retry in 605.651348ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
16892024-08-23T21:56:09.873ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb") failed: Property not found. retry in 1.245932285s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
16902024-08-23T21:56:09.981ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657") failed: Property not found. retry in 1.379421864s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
16912024-08-23T21:56:10.403ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd") failed: Property not found. retry in 540.05765ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
16922024-08-23T21:56:10.412ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855") failed: Property not found. retry in 1.116438506s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
16932024-08-23T21:56:10.557ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8") failed: Property not found. retry in 1.3272214s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
16942024-08-23T21:56:10.952ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd") failed: Property not found. retry in 1.072808117s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
16952024-08-23T21:56:11.130ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb") failed: Property not found. retry in 751.287348ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
16962024-08-23T21:56:11.370ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657") failed: Property not found. retry in 656.992722ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
16972024-08-23T21:56:11.538ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855") failed: Property not found. retry in 1.006954262s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
16982024-08-23T21:56:11.892ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb") failed: Property not found. retry in 1.268649132s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
16992024-08-23T21:56:11.900ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8") failed: Property not found. retry in 1.3775573s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
17002024-08-23T21:56:12.035ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd") failed: Property not found. retry in 1.062440756s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
17012024-08-23T21:56:12.043ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657") failed: Property not found. retry in 591.472402ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
17022024-08-23T21:56:12.555ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855") failed: Property not found. retry in 1.431071748s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
17032024-08-23T21:56:12.643ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657") failed: Property not found. retry in 970.93436ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
17042024-08-23T21:56:13.107ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd") failed: Property not found. retry in 963.612637ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
17052024-08-23T21:56:13.169ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb") failed: Property not found. retry in 799.026933ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
17062024-08-23T21:56:13.287ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8") failed: Property not found. retry in 1.413538194s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
17072024-08-23T21:56:13.624ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657") failed: Property not found. retry in 867.325799ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
17082024-08-23T21:56:13.978ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb") failed: Property not found. retry in 1.244092491s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
17092024-08-23T21:56:13.995ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855") failed: Property not found. retry in 725.756436ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
17102024-08-23T21:56:14.081ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd") failed: Property not found. retry in 1.167890024s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
17112024-08-23T21:56:14.501ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657") failed: Property not found. retry in 1.029512772s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
17122024-08-23T21:56:14.711ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8") failed: Property not found. retry in 1.179248907s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
17132024-08-23T21:56:14.730ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855") failed: Property not found. retry in 757.799437ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
17142024-08-23T21:56:15.231ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb") failed: Property not found. retry in 1.13669134s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
17152024-08-23T21:56:15.259ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd") failed: Property not found. retry in 1.18995371s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
17162024-08-23T21:56:15.500ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855") failed: Property not found. retry in 697.797982ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
17172024-08-23T21:56:15.539ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657") failed: Property not found. retry in 1.169732026s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
17182024-08-23T21:56:15.897ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8") failed: Property not found. retry in 1.393548775s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
17192024-08-23T21:56:16.206ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855") failed: Property not found. retry in 1.379984691s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
17202024-08-23T21:56:16.377ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb") failed: Property not found. retry in 830.063408ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
17212024-08-23T21:56:16.458ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd") failed: Property not found. retry in 1.115078673s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
17222024-08-23T21:56:16.717ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657") failed: Property not found. retry in 1.109478709s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
17232024-08-23T21:56:17.216ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb") failed: Property not found. retry in 1.459894894s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
17242024-08-23T21:56:17.301ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8") failed: Property not found. retry in 1.042558891s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
17252024-08-23T21:56:17.581ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd") failed: Property not found. retry in 1.274443555s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
17262024-08-23T21:56:17.595ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855") failed: Property not found. retry in 807.417029ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
17272024-08-23T21:56:17.836ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657") failed: Property not found. retry in 1.097634164s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
17282024-08-23T21:56:18.353ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8") failed: Property not found. retry in 1.415237383s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
17292024-08-23T21:56:18.412ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855") failed: Property not found. retry in 1.15226554s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
17302024-08-23T21:56:18.685ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb") failed: Property not found. retry in 1.462719987s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
17312024-08-23T21:56:18.865ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd") failed: Property not found. retry in 698.138862ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
17322024-08-23T21:56:18.943ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657") failed: Property not found. retry in 1.17200926s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
17332024-08-23T21:56:19.572ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd") failed: Property not found. retry in 987.208553ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
17342024-08-23T21:56:19.579ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855") failed: Property not found. retry in 685.21146ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
17352024-08-23T21:56:19.779ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8") failed: Property not found. retry in 1.038371171s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
17362024-08-23T21:56:19.788ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:42994
17372024-08-23T21:56:19.789ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 99 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:42994 req_id = 1f150717-74cd-4720-86ca-acead49ae127 response_code = 200 uri = /baseboard
17382024-08-23T21:56:20.127ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657") failed: Property not found. retry in 1.254561484s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
17392024-08-23T21:56:20.158ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb") failed: Property not found. retry in 517.112871ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
17402024-08-23T21:56:20.278ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855") failed: Property not found. retry in 1.488846355s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
17412024-08-23T21:56:20.569ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd") failed: Property not found. retry in 1.046634962s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
17422024-08-23T21:56:20.685ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb") failed: Property not found. retry in 1.001851316s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
17432024-08-23T21:56:20.829ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8") failed: Property not found. retry in 782.810401ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
17442024-08-23T21:56:21.391ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657") failed: Property not found. retry in 1.275512778s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
17452024-08-23T21:56:21.622ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8") failed: Property not found. retry in 1.289673227s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
17462024-08-23T21:56:21.633ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd") failed: Property not found. retry in 1.030839282s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
17472024-08-23T21:56:21.697ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb") failed: Property not found. retry in 504.530795ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
17482024-08-23T21:56:21.776ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855") failed: Property not found. retry in 900.87591ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
17492024-08-23T21:56:22.211ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb") failed: Property not found. retry in 829.501627ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
17502024-08-23T21:56:22.674ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd") failed: Property not found. retry in 1.043004627s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
17512024-08-23T21:56:22.681ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657") failed: Property not found. retry in 1.240633468s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
17522024-08-23T21:56:22.689ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855") failed: Property not found. retry in 669.4798ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
17532024-08-23T21:56:22.922ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8") failed: Property not found. retry in 925.317333ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
17542024-08-23T21:56:23.050ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb") failed: Property not found. retry in 1.479138277s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
17552024-08-23T21:56:23.368ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855") failed: Property not found. retry in 701.76019ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
17562024-08-23T21:56:23.726ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd") failed: Property not found. retry in 539.556294ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
17572024-08-23T21:56:23.857ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8") failed: Property not found. retry in 751.45629ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
17582024-08-23T21:56:23.932ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657") failed: Property not found. retry in 515.870818ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
17592024-08-23T21:56:24.080ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855") failed: Property not found. retry in 778.213932ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
17602024-08-23T21:56:24.275ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd") failed: Property not found. retry in 629.839682ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
17612024-08-23T21:56:24.459ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657") failed: Property not found. retry in 503.58354ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
17622024-08-23T21:56:24.540ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb") failed: Property not found. retry in 1.004601099s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
17632024-08-23T21:56:24.619ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8") failed: Property not found. retry in 1.453245078s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
17642024-08-23T21:56:24.970ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd") failed: Property not found. retry in 631.955677ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
17652024-08-23T21:56:26.097ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3245 zone = oxz_cockroachdb_5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657
17662024-08-23T21:56:26.097ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3245 zone = oxz_cockroachdb_32a152fc-12d5-4442-9ec7-44167f70e8eb
17672024-08-23T21:56:26.097ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3245 zone = oxz_cockroachdb_3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855
17682024-08-23T21:56:26.097ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3245 zone = oxz_cockroachdb_a32a4cfc-b6ee-416e-8738-c10a4146b4f8
17692024-08-23T21:56:26.097ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3245 zone = oxz_cockroachdb_5749b096-fa59-4034-bf5c-f4e555b65bfd
17702024-08-23T21:56:26.097ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
17712024-08-23T21:56:26.098ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
17722024-08-23T21:56:26.098ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 24943214 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49912 req_id = a4a5c932-0505-48aa-8dee-560a2c1e6dfd response_code = 204 uri = /omicron-zones
17732024-08-23T21:56:26.133ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:57137
17742024-08-23T21:56:26.164ZINFOSledAgent (ServiceManager): Initializing CRDB Cluster - sending request to [fd00:1122:3344:101::4]:32221 file = sled-agent/src/services.rs:3681
17752024-08-23T21:56:37.681ZINFOSledAgent (ServiceManager): Formatting CRDB file = sled-agent/src/services.rs:3699
17762024-08-23T21:56:41.695ZINFOSledAgent (ServiceManager): Formatting CRDB - Completed file = sled-agent/src/services.rs:3722
17772024-08-23T21:56:41.695ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 15561941 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:57137 req_id = b380356e-5e95-4513-9de6-7581e53fc039 response_code = 204 uri = /cockroachdb
17782024-08-23T21:56:41.735ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: bd1e844a-019f-4d2c-af1c-ec9d27991436, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 07617dac-1d06-4704-81b5-9345e9dd66a9, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: d738a4fe-561c-4114-8fad-21f02e2dba0e, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: cfa27805-1454-49c2-aa13-243fa7ea6adb, kind: Service { id: d738a4fe-561c-4114-8fad-21f02e2dba0e }, name: Name("ntp-d738a4fe-561c-4114-8fad-21f02e2dba0e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 151, 153])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1db57e91-9837-462c-9f2c-1b4fca780b7e, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 2a9b2cf7-2fb3-4597-871e-44332b4d82f2, kind: Service { id: 1db57e91-9837-462c-9f2c-1b4fca780b7e }, name: Name("external-dns-1db57e91-9837-462c-9f2c-1b4fca780b7e"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 231, 24])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 5d1239c9-696e-4616-a2a6-82968e91a2e1, kind: Service { id: b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac }, name: Name("external-dns-b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 184])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 4e26237f-8cdc-4da1-b44a-9b00d20bd873, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 00000426-1f68-402a-a60e-4fa2f73c115f, kind: Service { id: 4e26237f-8cdc-4da1-b44a-9b00d20bd873 }, name: Name("nexus-4e26237f-8cdc-4da1-b44a-9b00d20bd873"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 190, 168])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 2c94012a-643d-4f61-8185-c8786e00e320, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: b2284d48-b022-4af4-a2f6-3a691122eb62, kind: Service { id: 2c94012a-643d-4f61-8185-c8786e00e320 }, name: Name("nexus-2c94012a-643d-4f61-8185-c8786e00e320"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 253, 42])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: f9a2f81c-00cf-49a5-a96e-1c4dfeba1728, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 5cf061ea-987b-4986-8378-702c868c90c5, kind: Service { id: f9a2f81c-00cf-49a5-a96e-1c4dfeba1728 }, name: Name("nexus-f9a2f81c-00cf-49a5-a96e-1c4dfeba1728"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 196, 138])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: ba8b1177-03a3-49b6-a6ee-b364652c50c9, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: e1a016b1-16ad-48d4-a26f-2e916f51bc80 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8471ffba-0f68-4241-be3b-68581f848fb7, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: d9421cb5-f5bb-43e4-a2f0-337cff4f1100, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 8343e6aa-08fb-4b4a-bf94-a818d66d575d, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 7ac03450-9a27-4ec4-9669-9eb798da4419 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 1bb815f5-1d70-43b2-a7b7-1bb132d87fd6, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: dad70eae-b83f-4c3a-a32e-2a6a8946803c (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 92744278-0a6d-41bc-a502-b694867741c6, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3dafd3b1-1230-4db3-9961-2a4d53cc3a37, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External } } } }, OmicronZoneConfig { id: 66dd6deb-9c39-47ef-9f1c-98fea2467257, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External } } } }, OmicronZoneConfig { id: 9c2bbea7-8b8c-4d83-9618-26c0848936eb, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External } } } }, OmicronZoneConfig { id: 914d3516-8bfd-4006-87d8-91481730f9c7, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 2bda41aa-8135-413a-853b-44e7fc746744, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: dad70eae-b83f-4c3a-a32e-2a6a8946803c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dad70eae-b83f-4c3a-a32e-2a6a8946803c (zpool), kind: External } } } }, OmicronZoneConfig { id: 02433f6a-f8a9-4f05-af1b-be59dfd00a41, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External } } } }, OmicronZoneConfig { id: 83653856-00c4-44fe-b4df-ad99e78af473, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: e1a016b1-16ad-48d4-a26f-2e916f51bc80 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e1a016b1-16ad-48d4-a26f-2e916f51bc80 (zpool), kind: External } } } }, OmicronZoneConfig { id: 581275cf-e6a7-4ef5-ad1e-36b1379b2a18, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 7ac03450-9a27-4ec4-9669-9eb798da4419 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ac03450-9a27-4ec4-9669-9eb798da4419 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:440 sled_address = [fd00:1122:3344:101::1]:12345
17792024-08-23T21:56:41.736ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:55451
17802024-08-23T21:56:41.736ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
17812024-08-23T21:56:41.736ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: bd1e844a-019f-4d2c-af1c-ec9d27991436, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: InternalDns }, responder: ... }
17822024-08-23T21:56:41.767ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
17832024-08-23T21:56:41.767ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 60c8c02e-6e6f-4e1a-a162-ca6470555313, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: InternalDns }, responder: ... }
17842024-08-23T21:56:41.791ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
17852024-08-23T21:56:41.791ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 07617dac-1d06-4704-81b5-9345e9dd66a9, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: InternalDns }, responder: ... }
17862024-08-23T21:56:41.813ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
17872024-08-23T21:56:41.813ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5c4d7b3b-b9a4-4c5d-9fa8-057fdf1ca657, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17882024-08-23T21:56:41.834ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
17892024-08-23T21:56:41.834ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 32a152fc-12d5-4442-9ec7-44167f70e8eb, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17902024-08-23T21:56:41.854ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
17912024-08-23T21:56:41.854ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: a32a4cfc-b6ee-416e-8738-c10a4146b4f8, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17922024-08-23T21:56:41.876ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
17932024-08-23T21:56:41.876ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 5749b096-fa59-4034-bf5c-f4e555b65bfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17942024-08-23T21:56:41.897ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:406
17952024-08-23T21:56:41.897ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3a0f6ecf-1b05-4bf6-8a0a-f7e485f52855, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17962024-08-23T21:56:41.919ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1db57e91-9837-462c-9f2c-1b4fca780b7e, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
17972024-08-23T21:56:41.919ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 1db57e91-9837-462c-9f2c-1b4fca780b7e, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
17982024-08-23T21:56:41.998ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:406
17992024-08-23T21:56:41.998ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
18002024-08-23T21:56:42.085ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8471ffba-0f68-4241-be3b-68581f848fb7, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:406
18012024-08-23T21:56:42.085ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 8471ffba-0f68-4241-be3b-68581f848fb7, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
18022024-08-23T21:56:42.162ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 92744278-0a6d-41bc-a502-b694867741c6, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:406
18032024-08-23T21:56:42.162ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 92744278-0a6d-41bc-a502-b694867741c6, dataset_name: DatasetName { pool_name: ZpoolName { id: c4828395-89b1-4653-93a7-55b242871c48 (zpool), kind: External }, kind: Crucible }, responder: ... }
18042024-08-23T21:56:42.241ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3dafd3b1-1230-4db3-9961-2a4d53cc3a37, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:406
18052024-08-23T21:56:42.241ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 3dafd3b1-1230-4db3-9961-2a4d53cc3a37, dataset_name: DatasetName { pool_name: ZpoolName { id: c4fe4e01-40de-4c9e-857a-e6870bd84153 (zpool), kind: External }, kind: Crucible }, responder: ... }
18062024-08-23T21:56:42.316ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 66dd6deb-9c39-47ef-9f1c-98fea2467257, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:406
18072024-08-23T21:56:42.316ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 66dd6deb-9c39-47ef-9f1c-98fea2467257, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bb3d35c-cbec-445b-9b3a-58337934f8de (zpool), kind: External }, kind: Crucible }, responder: ... }
18082024-08-23T21:56:42.396ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9c2bbea7-8b8c-4d83-9618-26c0848936eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:406
18092024-08-23T21:56:42.396ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 9c2bbea7-8b8c-4d83-9618-26c0848936eb, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b92f88-0c07-4732-953e-9cac63715495 (zpool), kind: External }, kind: Crucible }, responder: ... }
18102024-08-23T21:56:42.471ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 914d3516-8bfd-4006-87d8-91481730f9c7, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:406
18112024-08-23T21:56:42.471ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 914d3516-8bfd-4006-87d8-91481730f9c7, dataset_name: DatasetName { pool_name: ZpoolName { id: 44ff68f6-909f-4df8-8e90-d68ca84323b3 (zpool), kind: External }, kind: Crucible }, responder: ... }
18122024-08-23T21:56:42.550ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2bda41aa-8135-413a-853b-44e7fc746744, dataset_name: DatasetName { pool_name: ZpoolName { id: dad70eae-b83f-4c3a-a32e-2a6a8946803c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:406
18132024-08-23T21:56:42.550ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 2bda41aa-8135-413a-853b-44e7fc746744, dataset_name: DatasetName { pool_name: ZpoolName { id: dad70eae-b83f-4c3a-a32e-2a6a8946803c (zpool), kind: External }, kind: Crucible }, responder: ... }
18142024-08-23T21:56:42.615ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 02433f6a-f8a9-4f05-af1b-be59dfd00a41, dataset_name: DatasetName { pool_name: ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:406
18152024-08-23T21:56:42.616ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 02433f6a-f8a9-4f05-af1b-be59dfd00a41, dataset_name: DatasetName { pool_name: ZpoolName { id: ec50006e-a21c-4820-b439-48f6d6b60c97 (zpool), kind: External }, kind: Crucible }, responder: ... }
18162024-08-23T21:56:42.679ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83653856-00c4-44fe-b4df-ad99e78af473, dataset_name: DatasetName { pool_name: ZpoolName { id: e1a016b1-16ad-48d4-a26f-2e916f51bc80 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:406
18172024-08-23T21:56:42.679ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 83653856-00c4-44fe-b4df-ad99e78af473, dataset_name: DatasetName { pool_name: ZpoolName { id: e1a016b1-16ad-48d4-a26f-2e916f51bc80 (zpool), kind: External }, kind: Crucible }, responder: ... }
18182024-08-23T21:56:42.746ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 581275cf-e6a7-4ef5-ad1e-36b1379b2a18, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ac03450-9a27-4ec4-9669-9eb798da4419 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:406
18192024-08-23T21:56:42.746ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:773 request = NewFilesystemRequest { dataset_id: 581275cf-e6a7-4ef5-ad1e-36b1379b2a18, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ac03450-9a27-4ec4-9669-9eb798da4419 (zpool), kind: External }, kind: Crucible }, responder: ... }
18202024-08-23T21:56:42.812ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
18212024-08-23T21:56:42.812ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
18222024-08-23T21:56:42.812ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
18232024-08-23T21:56:42.813ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
18242024-08-23T21:56:42.940ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
18252024-08-23T21:56:42.940ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
18262024-08-23T21:56:42.957ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
18272024-08-23T21:56:42.957ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
18282024-08-23T21:56:42.957ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
18292024-08-23T21:56:42.990ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
18302024-08-23T21:56:42.990ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
18312024-08-23T21:56:42.990ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
18322024-08-23T21:56:42.990ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
18332024-08-23T21:56:42.992ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
18342024-08-23T21:56:43.146ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
18352024-08-23T21:56:43.147ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
18362024-08-23T21:56:43.147ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
18372024-08-23T21:56:43.207ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
18382024-08-23T21:56:43.207ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
18392024-08-23T21:56:43.207ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
18402024-08-23T21:56:43.207ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
18412024-08-23T21:56:43.207ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
18422024-08-23T21:56:43.207ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
18432024-08-23T21:56:43.207ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7 file = illumos-utils/src/zone.rs:316
18442024-08-23T21:56:43.214ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6 file = illumos-utils/src/zone.rs:316
18452024-08-23T21:56:43.221ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6 file = illumos-utils/src/zone.rs:316
18462024-08-23T21:56:43.228ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9 file = illumos-utils/src/zone.rs:316
18472024-08-23T21:56:43.233ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37 file = illumos-utils/src/zone.rs:316
18482024-08-23T21:56:43.239ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100 file = illumos-utils/src/zone.rs:316
18492024-08-23T21:56:43.413ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7 file = illumos-utils/src/zone.rs:354
18502024-08-23T21:56:43.418ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6 file = illumos-utils/src/zone.rs:354
18512024-08-23T21:56:43.424ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6 file = illumos-utils/src/zone.rs:354
18522024-08-23T21:56:43.428ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9 file = illumos-utils/src/zone.rs:354
18532024-08-23T21:56:43.432ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37 file = illumos-utils/src/zone.rs:354
18542024-08-23T21:56:43.437ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100 file = illumos-utils/src/zone.rs:354
18552024-08-23T21:56:43.443ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257 file = illumos-utils/src/zone.rs:316
18562024-08-23T21:56:43.448ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb file = illumos-utils/src/zone.rs:316
18572024-08-23T21:56:43.454ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744 file = illumos-utils/src/zone.rs:316
18582024-08-23T21:56:43.460ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18 file = illumos-utils/src/zone.rs:316
18592024-08-23T21:56:43.466ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d file = illumos-utils/src/zone.rs:316
18602024-08-23T21:56:43.473ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_clickhouse_8471ffba-0f68-4241-be3b-68581f848fb7 file = illumos-utils/src/zone.rs:316
18612024-08-23T21:56:43.480ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473 file = illumos-utils/src/zone.rs:316
18622024-08-23T21:56:43.487ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41 file = illumos-utils/src/zone.rs:316
18632024-08-23T21:56:43.541ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
18642024-08-23T21:56:43.542ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257 file = illumos-utils/src/zone.rs:354
18652024-08-23T21:56:43.550ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb file = illumos-utils/src/zone.rs:354
18662024-08-23T21:56:43.559ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744 file = illumos-utils/src/zone.rs:354
18672024-08-23T21:56:43.568ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18 file = illumos-utils/src/zone.rs:354
18682024-08-23T21:56:43.577ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d file = illumos-utils/src/zone.rs:354
18692024-08-23T21:56:43.585ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_clickhouse_8471ffba-0f68-4241-be3b-68581f848fb7 file = illumos-utils/src/zone.rs:354
18702024-08-23T21:56:43.595ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473 file = illumos-utils/src/zone.rs:354
18712024-08-23T21:56:43.606ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41 file = illumos-utils/src/zone.rs:354
18722024-08-23T21:56:43.693ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
18732024-08-23T21:56:43.882ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
18742024-08-23T21:56:43.943ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
18752024-08-23T21:56:44.004ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
18762024-08-23T21:56:44.008ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
18772024-08-23T21:56:44.008ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
18782024-08-23T21:56:44.126ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte1", ip: 172.30.1.5, mac: MacAddr6([168, 64, 37, 255, 231, 24]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
18792024-08-23T21:56:44.127ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1235 zone_type = external_dns
18802024-08-23T21:56:44.128ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
18812024-08-23T21:56:44.128ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
18822024-08-23T21:56:44.186ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte2", ip: 172.30.2.7, mac: MacAddr6([168, 64, 37, 255, 196, 138]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
18832024-08-23T21:56:44.186ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1235 zone_type = nexus
18842024-08-23T21:56:44.188ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
18852024-08-23T21:56:44.188ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
18862024-08-23T21:56:44.247ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte3", ip: 172.30.1.6, mac: MacAddr6([168, 64, 37, 255, 163, 184]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
18872024-08-23T21:56:44.247ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1235 zone_type = external_dns
18882024-08-23T21:56:44.248ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
18892024-08-23T21:56:44.248ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
18902024-08-23T21:56:44.300ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte4", ip: 172.30.2.6, mac: MacAddr6([168, 64, 37, 255, 253, 42]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
18912024-08-23T21:56:44.300ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1235 zone_type = nexus
18922024-08-23T21:56:44.302ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
18932024-08-23T21:56:44.302ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
18942024-08-23T21:56:44.358ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte5", ip: 172.30.2.5, mac: MacAddr6([168, 64, 37, 255, 190, 168]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
18952024-08-23T21:56:44.358ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1235 zone_type = nexus
18962024-08-23T21:56:44.358ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:83
18972024-08-23T21:56:44.358ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:90
18982024-08-23T21:56:44.358ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:83
18992024-08-23T21:56:44.358ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:90
19002024-08-23T21:56:44.359ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:83
19012024-08-23T21:56:44.359ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:90
19022024-08-23T21:56:44.359ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:83
19032024-08-23T21:56:44.359ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:90
19042024-08-23T21:56:44.384ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:118
19052024-08-23T21:56:44.386ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
19062024-08-23T21:56:44.431ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:118
19072024-08-23T21:56:44.434ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:118
19082024-08-23T21:56:44.435ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
19092024-08-23T21:56:44.438ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:118
19102024-08-23T21:56:44.438ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
19112024-08-23T21:56:44.441ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
19122024-08-23T21:56:44.441ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:83
19132024-08-23T21:56:44.441ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:90
19142024-08-23T21:56:44.587ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:118
19152024-08-23T21:56:44.591ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
19162024-08-23T21:56:44.591ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e file = illumos-utils/src/zone.rs:316
19172024-08-23T21:56:44.604ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728 file = illumos-utils/src/zone.rs:316
19182024-08-23T21:56:44.615ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac file = illumos-utils/src/zone.rs:316
19192024-08-23T21:56:44.627ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320 file = illumos-utils/src/zone.rs:316
19202024-08-23T21:56:44.685ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e file = illumos-utils/src/zone.rs:354
19212024-08-23T21:56:44.694ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728 file = illumos-utils/src/zone.rs:354
19222024-08-23T21:56:44.714ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac file = illumos-utils/src/zone.rs:354
19232024-08-23T21:56:44.739ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320 file = illumos-utils/src/zone.rs:354
19242024-08-23T21:56:44.756ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873 file = illumos-utils/src/zone.rs:316
19252024-08-23T21:56:44.810ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873 file = illumos-utils/src/zone.rs:354
19262024-08-23T21:56:46.821ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService12'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::11'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::11'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19272024-08-23T21:56:46.822ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6
19282024-08-23T21:56:46.849ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService15'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::f'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::f'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19292024-08-23T21:56:46.849ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100
19302024-08-23T21:56:46.965ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService11'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::12'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_c4828395-89b1-4653-93a7-55b242871c48/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::12'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='ec54e84a-d12c-4009-ac5c-2cd66ff355e1'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19312024-08-23T21:56:46.965ZINFOSledAgent (ServiceManager): Booting oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6
19322024-08-23T21:56:46.979ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService10'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::16'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_44ff68f6-909f-4df8-8e90-d68ca84323b3/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::16'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='16c50261-346a-4ae1-866c-1794aa3d0030'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19332024-08-23T21:56:46.979ZINFOSledAgent (ServiceManager): Booting oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7
19342024-08-23T21:56:46.988ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService14'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::13'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_c4fe4e01-40de-4c9e-857a-e6870bd84153/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::13'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='863a3e1a-b682-4c85-a361-7d6c696f64a9'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19352024-08-23T21:56:46.989ZINFOSledAgent (ServiceManager): Booting oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37
19362024-08-23T21:56:47.000ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService16'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::14'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_0bb3d35c-cbec-445b-9b3a-58337934f8de/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::14'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='3ae8de59-6039-43f7-90a7-1ce3077d1680'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19372024-08-23T21:56:47.000ZINFOSledAgent (ServiceManager): Booting oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257
19382024-08-23T21:56:47.047ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService20'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::10'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::10'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19392024-08-23T21:56:47.047ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d
19402024-08-23T21:56:47.110ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService17'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::15'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_44b92f88-0c07-4732-953e-9cac63715495/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::15'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='ad15ca28-15d1-4ad6-b348-3adcd0578569'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19412024-08-23T21:56:47.110ZINFOSledAgent (ServiceManager): Booting oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb
19422024-08-23T21:56:47.121ZINFOSledAgent (ServiceManager): Profile for oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService13'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::d'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/oximeter"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="id" value='ba8b1177-03a3-49b6-a6ee-b364652c50c9'/> <propval type="astring" name="address" value='[fd00:1122:3344:101::d]:12223'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19432024-08-23T21:56:47.121ZINFOSledAgent (ServiceManager): Booting oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9
19442024-08-23T21:56:47.132ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService23'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::18'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_ec50006e-a21c-4820-b439-48f6d6b60c97/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::18'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='f5964858-ddf5-4f8d-8bb0-342bfb9935da'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19452024-08-23T21:56:47.132ZINFOSledAgent (ServiceManager): Booting oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41
19462024-08-23T21:56:47.149ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService19'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_7ac03450-9a27-4ec4-9669-9eb798da4419/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1a'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='129890bb-058a-4b17-a14d-1cf8399fdfb5'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19472024-08-23T21:56:47.149ZINFOSledAgent (ServiceManager): Booting oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18
19482024-08-23T21:56:48.737ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6") failed: Property not found. retry in 44.432858ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6
19492024-08-23T21:56:54.395ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37") failed: Property not found. retry in 44.813711ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37
19502024-08-23T21:56:55.178ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7") failed: Property not found. retry in 52.169421ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7
19512024-08-23T21:56:55.197ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6") failed: Property not found. retry in 74.899339ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6
19522024-08-23T21:56:55.349ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6") failed: Property not found. retry in 141.344141ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6
19532024-08-23T21:56:55.449ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9") failed: Property not found. retry in 58.447375ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9
19542024-08-23T21:56:55.544ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37") failed: Property not found. retry in 89.49052ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37
19552024-08-23T21:56:55.702ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100") failed: Property not found. retry in 68.350564ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100
19562024-08-23T21:56:55.746ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257") failed: Property not found. retry in 42.853844ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257
19572024-08-23T21:56:55.777ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb") failed: Property not found. retry in 29.711969ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb
19582024-08-23T21:56:55.819ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService18'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::17'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_dad70eae-b83f-4c3a-a32e-2a6a8946803c/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::17'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='171036ff-7f50-48e6-b9a4-5ffa9a0f8403'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19592024-08-23T21:56:55.834ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18") failed: Property not found. retry in 61.168395ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18
19602024-08-23T21:56:55.860ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d") failed: Property not found. retry in 62.524802ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d
19612024-08-23T21:56:55.922ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService22'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::19'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_e1a016b1-16ad-48d4-a26f-2e916f51bc80/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::19'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='6ab6a6ee-c56d-48ed-817b-87ea9c6fc0f1'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19622024-08-23T21:56:55.952ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41") failed: Property not found. retry in 67.086924ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41
19632024-08-23T21:56:55.978ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7") failed: Property not found. retry in 131.64085ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7
19642024-08-23T21:56:56.002ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6") failed: Property not found. retry in 143.089513ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6
19652024-08-23T21:56:56.015ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6") failed: Property not found. retry in 165.003488ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6
19662024-08-23T21:56:56.028ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9") failed: Property not found. retry in 145.703664ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9
19672024-08-23T21:56:56.044ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37") failed: Property not found. retry in 134.524242ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37
19682024-08-23T21:56:56.071ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100") failed: Property not found. retry in 67.033322ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100
19692024-08-23T21:56:56.097ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257") failed: Property not found. retry in 130.355428ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257
19702024-08-23T21:56:56.111ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb") failed: Property not found. retry in 139.215243ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb
19712024-08-23T21:56:56.111ZINFOSledAgent (ServiceManager): Booting oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744
19722024-08-23T21:56:56.156ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18") failed: Property not found. retry in 128.53612ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18
19732024-08-23T21:56:56.179ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d") failed: Property not found. retry in 142.428939ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d
19742024-08-23T21:56:56.179ZINFOSledAgent (ServiceManager): Booting oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473
19752024-08-23T21:56:56.228ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41") failed: Property not found. retry in 56.79429ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41
19762024-08-23T21:56:56.244ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7") failed: Property not found. retry in 293.660846ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7
19772024-08-23T21:56:56.257ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6") failed: Property not found. retry in 224.004466ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6
19782024-08-23T21:56:56.270ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6") failed: Property not found. retry in 458.198024ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6
19792024-08-23T21:56:56.283ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9") failed: Property not found. retry in 210.532892ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9
19802024-08-23T21:56:56.295ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37") failed: Property not found. retry in 478.4974ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37
19812024-08-23T21:56:56.306ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100") failed: Property not found. retry in 267.678474ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100
19822024-08-23T21:56:56.317ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257") failed: Property not found. retry in 122.205956ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257
19832024-08-23T21:56:56.339ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb") failed: Property not found. retry in 291.713408ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb
19842024-08-23T21:56:56.352ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18") failed: Property not found. retry in 161.550424ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18
19852024-08-23T21:56:56.363ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d") failed: Property not found. retry in 298.107409ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d
19862024-08-23T21:56:56.374ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41") failed: Property not found. retry in 146.789262ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41
19872024-08-23T21:56:56.420ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744") failed: Property not found. retry in 73.542299ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744
19882024-08-23T21:56:56.431ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473") failed: Property not found. retry in 33.8274ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473
19892024-08-23T21:56:56.454ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257") failed: Property not found. retry in 295.947691ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257
19902024-08-23T21:56:56.476ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473") failed: Property not found. retry in 143.012309ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473
19912024-08-23T21:56:56.495ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6") failed: Property not found. retry in 261.668668ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6
19922024-08-23T21:56:56.510ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9") failed: Property not found. retry in 459.938941ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9
19932024-08-23T21:56:56.524ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744") failed: Property not found. retry in 122.332515ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744
19942024-08-23T21:56:56.538ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18") failed: Property not found. retry in 384.113412ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18
19952024-08-23T21:56:56.552ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41") failed: Property not found. retry in 558.976963ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41
19962024-08-23T21:56:56.567ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7") failed: Property not found. retry in 377.384523ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7
19972024-08-23T21:56:56.586ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100") failed: Property not found. retry in 508.246811ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100
19982024-08-23T21:56:56.633ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473") failed: Property not found. retry in 261.154502ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473
19992024-08-23T21:56:56.648ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb") failed: Property not found. retry in 579.539565ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb
20002024-08-23T21:56:56.661ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744") failed: Property not found. retry in 260.542441ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744
20012024-08-23T21:56:56.674ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d") failed: Property not found. retry in 251.326515ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d
20022024-08-23T21:56:56.744ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6") failed: Property not found. retry in 843.213939ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6
20032024-08-23T21:56:56.765ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257") failed: Property not found. retry in 1.179074894s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257
20042024-08-23T21:56:56.777ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6") failed: Property not found. retry in 654.722173ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6
20052024-08-23T21:56:56.789ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37") failed: Property not found. retry in 886.227842ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37
20062024-08-23T21:56:56.911ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473") failed: Property not found. retry in 290.057933ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473
20072024-08-23T21:56:56.913ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService24'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::8'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte1'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::8]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.5:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
20082024-08-23T21:56:56.914ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e
20092024-08-23T21:56:56.938ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744") failed: Property not found. retry in 401.897111ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744
20102024-08-23T21:56:56.952ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18") failed: Property not found. retry in 767.010846ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18
20112024-08-23T21:56:56.966ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d") failed: Property not found. retry in 1.071650793s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d
20122024-08-23T21:56:56.980ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7") failed: Property not found. retry in 540.074432ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7
20132024-08-23T21:56:56.994ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9") failed: Property not found. retry in 561.637428ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9
20142024-08-23T21:56:57.106ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100") failed: Property not found. retry in 1.02632558s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100
20152024-08-23T21:56:57.129ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41") failed: Property not found. retry in 912.03516ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41
20162024-08-23T21:56:57.174ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e") failed: Property not found. retry in 40.808154ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e
20172024-08-23T21:56:57.216ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473") failed: Property not found. retry in 1.144287566s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473
20182024-08-23T21:56:57.228ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e") failed: Property not found. retry in 94.970545ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e
20192024-08-23T21:56:57.240ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb") failed: Property not found. retry in 1.124099439s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb
20202024-08-23T21:56:57.335ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e") failed: Property not found. retry in 237.371496ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e
20212024-08-23T21:56:57.353ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744") failed: Property not found. retry in 1.039490117s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744
20222024-08-23T21:56:57.446ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6") failed: Property not found. retry in 931.218283ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6
20232024-08-23T21:56:57.533ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7") failed: Property not found. retry in 528.085985ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7
20242024-08-23T21:56:57.569ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9") failed: Property not found. retry in 925.199754ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9
20252024-08-23T21:56:57.586ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e") failed: Property not found. retry in 362.778061ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e
20262024-08-23T21:56:57.602ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6") failed: Property not found. retry in 550.573982ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6
20272024-08-23T21:56:57.689ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37") failed: Property not found. retry in 1.124971399s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37
20282024-08-23T21:56:57.732ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18") failed: Property not found. retry in 1.442245557s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18
20292024-08-23T21:56:57.957ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257") failed: Property not found. retry in 1.129348502s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257
20302024-08-23T21:56:57.970ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e") failed: Property not found. retry in 694.791646ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e
20312024-08-23T21:56:57.993ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService28'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte5'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
20322024-08-23T21:56:57.993ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService25'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::c'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte2'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
20332024-08-23T21:56:57.994ZINFOSledAgent (ServiceManager): Booting oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873
20342024-08-23T21:56:58.001ZINFOSledAgent (ServiceManager): Booting oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728
20352024-08-23T21:56:58.051ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d") failed: Property not found. retry in 677.617475ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d
20362024-08-23T21:56:58.062ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41") failed: Property not found. retry in 1.304425663s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41
20372024-08-23T21:56:58.075ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7") failed: Property not found. retry in 943.229194ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7
20382024-08-23T21:56:58.145ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100") failed: Property not found. retry in 1.436546632s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100
20392024-08-23T21:56:58.166ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6") failed: Property not found. retry in 1.228841576s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6
20402024-08-23T21:56:58.214ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873") failed: Property not found. retry in 60.887988ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873
20412024-08-23T21:56:58.224ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728") failed: Property not found. retry in 55.439424ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728
20422024-08-23T21:56:58.286ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873") failed: Property not found. retry in 105.867429ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873
20432024-08-23T21:56:58.295ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728") failed: Property not found. retry in 106.791528ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728
20442024-08-23T21:56:58.373ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473") failed: Property not found. retry in 513.766005ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473
20452024-08-23T21:56:58.383ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb") failed: Property not found. retry in 882.798579ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb
20462024-08-23T21:56:58.395ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6") failed: Property not found. retry in 759.85758ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6
20472024-08-23T21:56:58.405ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873") failed: Property not found. retry in 219.038706ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873
20482024-08-23T21:56:58.418ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744") failed: Property not found. retry in 712.821372ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744
20492024-08-23T21:56:58.429ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728") failed: Property not found. retry in 170.170348ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728
20502024-08-23T21:56:58.506ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9") failed: Property not found. retry in 1.088044382s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9
20512024-08-23T21:56:58.613ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728") failed: Property not found. retry in 312.676992ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728
20522024-08-23T21:56:58.639ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873") failed: Property not found. retry in 359.228543ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873
20532024-08-23T21:56:58.677ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e") failed: Property not found. retry in 1.15466898s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e
20542024-08-23T21:56:58.741ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d") failed: Property not found. retry in 1.477337458s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d
20552024-08-23T21:56:58.826ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37") failed: Property not found. retry in 715.262962ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37
20562024-08-23T21:56:58.898ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473") failed: Property not found. retry in 682.244ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473
20572024-08-23T21:56:58.935ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728") failed: Property not found. retry in 1.144646747s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728
20582024-08-23T21:56:59.010ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873") failed: Property not found. retry in 809.697437ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873
20592024-08-23T21:56:59.030ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7") failed: Property not found. retry in 784.306404ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7
20602024-08-23T21:56:59.098ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257") failed: Property not found. retry in 1.077028745s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257
20612024-08-23T21:56:59.142ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744") failed: Property not found. retry in 772.39999ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744
20622024-08-23T21:56:59.166ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6") failed: Property not found. retry in 1.436884671s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6
20632024-08-23T21:56:59.185ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18") failed: Property not found. retry in 1.175145417s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18
20642024-08-23T21:56:59.278ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb") failed: Property not found. retry in 546.503736ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb
20652024-08-23T21:56:59.380ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41") failed: Property not found. retry in 1.069923725s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41
20662024-08-23T21:56:59.406ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6") failed: Property not found. retry in 1.305471122s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6
20672024-08-23T21:56:59.554ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37") failed: Property not found. retry in 656.653776ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37
20682024-08-23T21:56:59.594ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473") failed: Property not found. retry in 1.258387701s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473
20692024-08-23T21:56:59.605ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100") failed: Property not found. retry in 1.403127027s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100
20702024-08-23T21:56:59.617ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9") failed: Property not found. retry in 1.028282801s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9
20712024-08-23T21:56:59.829ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7") failed: Property not found. retry in 987.112309ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7
20722024-08-23T21:56:59.842ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb") failed: Property not found. retry in 834.997202ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb
20732024-08-23T21:56:59.854ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873") failed: Property not found. retry in 1.406171506s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873
20742024-08-23T21:56:59.868ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e") failed: Property not found. retry in 698.356998ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e
20752024-08-23T21:56:59.929ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744") failed: Property not found. retry in 529.686296ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744
20762024-08-23T21:57:00.093ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728") failed: Property not found. retry in 1.091758997s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728
20772024-08-23T21:57:00.189ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257") failed: Property not found. retry in 1.117006387s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257
20782024-08-23T21:57:00.224ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37") failed: Property not found. retry in 1.397439517s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37
20792024-08-23T21:57:00.236ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d") failed: Property not found. retry in 894.891002ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d
20802024-08-23T21:57:00.374ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18") failed: Property not found. retry in 849.583845ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18
20812024-08-23T21:57:00.459ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41") failed: Property not found. retry in 1.187037163s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41
20822024-08-23T21:57:00.468ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744") failed: Property not found. retry in 596.52734ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744
20832024-08-23T21:57:00.576ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e") failed: Property not found. retry in 1.382906434s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e
20842024-08-23T21:57:00.613ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6") failed: Property not found. retry in 1.085587954s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6
20852024-08-23T21:57:00.656ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9") failed: Property not found. retry in 789.786884ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9
20862024-08-23T21:57:00.687ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb") failed: Property not found. retry in 642.742441ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb
20872024-08-23T21:57:00.721ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6") failed: Property not found. retry in 873.847722ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6
20882024-08-23T21:57:00.826ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7") failed: Property not found. retry in 1.436609456s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7
20892024-08-23T21:57:00.861ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473") failed: Property not found. retry in 753.162774ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473
20902024-08-23T21:57:01.017ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100") failed: Property not found. retry in 599.995479ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100
20912024-08-23T21:57:01.074ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744") failed: Property not found. retry in 534.590173ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744
20922024-08-23T21:57:01.140ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d") failed: Property not found. retry in 1.346970122s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d
20932024-08-23T21:57:01.195ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728") failed: Property not found. retry in 1.45975504s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728
20942024-08-23T21:57:01.232ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18") failed: Property not found. retry in 1.050654824s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18
20952024-08-23T21:57:01.272ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873") failed: Property not found. retry in 1.161935992s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873
20962024-08-23T21:57:01.337ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257") failed: Property not found. retry in 1.0456646s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257
20972024-08-23T21:57:01.353ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb") failed: Property not found. retry in 1.360801899s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb
20982024-08-23T21:57:01.459ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9") failed: Property not found. retry in 1.177302205s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9
20992024-08-23T21:57:01.608ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6") failed: Property not found. retry in 1.291872425s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6
21002024-08-23T21:57:01.619ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744") failed: Property not found. retry in 1.257490302s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744
21012024-08-23T21:57:01.627ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100") failed: Property not found. retry in 1.181565814s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100
21022024-08-23T21:57:01.636ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473") failed: Property not found. retry in 952.911951ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473
21032024-08-23T21:57:01.646ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37") failed: Property not found. retry in 1.32745199s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37
21042024-08-23T21:57:01.658ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41") failed: Property not found. retry in 664.181405ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41
21052024-08-23T21:57:01.714ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6") failed: Property not found. retry in 1.051603106s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6
21062024-08-23T21:57:01.971ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e") failed: Property not found. retry in 560.135893ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e
21072024-08-23T21:57:02.060ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService26'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::9'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte3'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::9]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.6:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
21082024-08-23T21:57:02.061ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac
21092024-08-23T21:57:02.276ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7") failed: Property not found. retry in 756.83382ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7
21102024-08-23T21:57:02.296ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18") failed: Property not found. retry in 1.230950972s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18
21112024-08-23T21:57:02.336ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41") failed: Property not found. retry in 1.3524135s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41
21122024-08-23T21:57:02.384ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac") failed: Property not found. retry in 52.217109ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac
21132024-08-23T21:57:02.395ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257") failed: Property not found. retry in 1.461253796s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257
21142024-08-23T21:57:02.447ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873") failed: Property not found. retry in 607.427304ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873
21152024-08-23T21:57:02.456ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac") failed: Property not found. retry in 112.905715ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac
21162024-08-23T21:57:02.500ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d") failed: Property not found. retry in 1.057177066s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d
21172024-08-23T21:57:02.544ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e") failed: Property not found. retry in 1.493872832s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e
21182024-08-23T21:57:02.580ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac") failed: Property not found. retry in 205.789895ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac
21192024-08-23T21:57:02.600ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473") failed: Property not found. retry in 1.447352865s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473
21202024-08-23T21:57:02.646ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9") failed: Property not found. retry in 1.465671937s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9
21212024-08-23T21:57:02.665ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728") failed: Property not found. retry in 1.148377585s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728
21222024-08-23T21:57:02.725ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb") failed: Property not found. retry in 1.130969384s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb
21232024-08-23T21:57:02.780ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6") failed: Property not found. retry in 579.387719ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6
21242024-08-23T21:57:02.796ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac") failed: Property not found. retry in 529.309842ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac
21252024-08-23T21:57:02.822ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100") failed: Property not found. retry in 1.150051075s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100
21262024-08-23T21:57:02.896ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744") failed: Property not found. retry in 707.361368ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744
21272024-08-23T21:57:02.913ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6") failed: Property not found. retry in 968.343548ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6
21282024-08-23T21:57:02.983ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37") failed: Property not found. retry in 1.280458646s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37
21292024-08-23T21:57:03.042ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7") failed: Property not found. retry in 1.288992321s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7
21302024-08-23T21:57:03.044ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService27'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte4'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
21312024-08-23T21:57:03.044ZINFOSledAgent (ServiceManager): Booting oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320
21322024-08-23T21:57:03.065ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873") failed: Property not found. retry in 1.290851971s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873
21332024-08-23T21:57:03.197ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320") failed: Property not found. retry in 74.749438ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320
21342024-08-23T21:57:03.281ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320") failed: Property not found. retry in 113.191613ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320
21352024-08-23T21:57:03.335ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac") failed: Property not found. retry in 537.745829ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac
21362024-08-23T21:57:03.370ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6") failed: Property not found. retry in 1.024906976s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6
21372024-08-23T21:57:03.408ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320") failed: Property not found. retry in 228.222103ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320
21382024-08-23T21:57:03.537ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18") failed: Property not found. retry in 1.215361937s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18
21392024-08-23T21:57:03.567ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d") failed: Property not found. retry in 1.178149667s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d
21402024-08-23T21:57:03.613ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744") failed: Property not found. retry in 602.076772ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744
21412024-08-23T21:57:03.645ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320") failed: Property not found. retry in 449.396612ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320
21422024-08-23T21:57:03.698ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41") failed: Property not found. retry in 1.181484743s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41
21432024-08-23T21:57:03.823ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728") failed: Property not found. retry in 504.440473ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728
21442024-08-23T21:57:03.866ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb") failed: Property not found. retry in 659.251782ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb
21452024-08-23T21:57:03.874ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257") failed: Property not found. retry in 1.176473734s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257
21462024-08-23T21:57:03.882ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac") failed: Property not found. retry in 730.295413ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac
21472024-08-23T21:57:03.890ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6") failed: Property not found. retry in 951.351039ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6
21482024-08-23T21:57:03.982ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100") failed: Property not found. retry in 940.578068ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100
21492024-08-23T21:57:04.047ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e") failed: Property not found. retry in 783.348553ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e
21502024-08-23T21:57:04.056ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473") failed: Property not found. retry in 560.483455ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473
21512024-08-23T21:57:04.105ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320") failed: Property not found. retry in 617.667682ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320
21522024-08-23T21:57:04.120ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9") failed: Property not found. retry in 1.057265152s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9
21532024-08-23T21:57:04.225ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744") failed: Property not found. retry in 770.662492ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744
21542024-08-23T21:57:04.271ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37") failed: Property not found. retry in 779.257811ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37
21552024-08-23T21:57:04.334ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728") failed: Property not found. retry in 1.188836519s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728
21562024-08-23T21:57:04.341ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7") failed: Property not found. retry in 1.189925694s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7
21572024-08-23T21:57:04.365ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873") failed: Property not found. retry in 1.476205283s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873
21582024-08-23T21:57:04.402ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6") failed: Property not found. retry in 1.35900322s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6
21592024-08-23T21:57:04.534ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb") failed: Property not found. retry in 585.816331ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb
21602024-08-23T21:57:04.621ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac") failed: Property not found. retry in 1.165364824s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac
21612024-08-23T21:57:04.630ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473") failed: Property not found. retry in 1.093664176s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473
21622024-08-23T21:57:04.731ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320") failed: Property not found. retry in 984.882059ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320
21632024-08-23T21:57:04.753ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d") failed: Property not found. retry in 827.386796ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d
21642024-08-23T21:57:04.762ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18") failed: Property not found. retry in 1.216343275s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18
21652024-08-23T21:57:04.840ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e") failed: Property not found. retry in 1.405282241s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e
21662024-08-23T21:57:04.850ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6") failed: Property not found. retry in 1.269891922s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6
21672024-08-23T21:57:04.889ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41") failed: Property not found. retry in 786.233189ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41
21682024-08-23T21:57:04.931ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100") failed: Property not found. retry in 923.083571ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100
21692024-08-23T21:57:05.004ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744") failed: Property not found. retry in 1.335057307s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744
21702024-08-23T21:57:05.059ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37") failed: Property not found. retry in 1.10363099s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37
21712024-08-23T21:57:05.067ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257") failed: Property not found. retry in 836.727787ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257
21722024-08-23T21:57:05.128ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb") failed: Property not found. retry in 839.434422ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb
21732024-08-23T21:57:05.187ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9") failed: Property not found. retry in 1.106412201s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9
21742024-08-23T21:57:05.532ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728") failed: Property not found. retry in 1.068295174s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728
21752024-08-23T21:57:05.540ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7") failed: Property not found. retry in 737.332826ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7
21762024-08-23T21:57:05.589ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d") failed: Property not found. retry in 1.254892634s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d
21772024-08-23T21:57:05.685ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41") failed: Property not found. retry in 1.019759977s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41
21782024-08-23T21:57:05.725ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320") failed: Property not found. retry in 618.520373ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320
21792024-08-23T21:57:05.733ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473") failed: Property not found. retry in 983.288287ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473
21802024-08-23T21:57:05.771ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6") failed: Property not found. retry in 1.242112295s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6
21812024-08-23T21:57:05.796ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac") failed: Property not found. retry in 1.161398074s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac
21822024-08-23T21:57:05.849ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873") failed: Property not found. retry in 547.901698ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873
21832024-08-23T21:57:05.864ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100") failed: Property not found. retry in 521.532961ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100
21842024-08-23T21:57:05.913ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257") failed: Property not found. retry in 783.728236ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257
21852024-08-23T21:57:05.977ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb") failed: Property not found. retry in 716.660353ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb
21862024-08-23T21:57:05.988ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18") failed: Property not found. retry in 1.220163039s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18
21872024-08-23T21:57:06.129ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6") failed: Property not found. retry in 1.44413171s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6
21882024-08-23T21:57:06.172ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37") failed: Property not found. retry in 824.291655ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37
21892024-08-23T21:57:06.256ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e") failed: Property not found. retry in 1.322493761s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e
21902024-08-23T21:57:06.285ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7") failed: Property not found. retry in 1.177979982s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7
21912024-08-23T21:57:06.301ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9") failed: Property not found. retry in 1.252200607s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9
21922024-08-23T21:57:06.346ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744") failed: Property not found. retry in 1.42787529s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744
21932024-08-23T21:57:06.353ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320") failed: Property not found. retry in 953.60322ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320
21942024-08-23T21:57:06.394ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100") failed: Property not found. retry in 626.929597ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100
21952024-08-23T21:57:06.405ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873") failed: Property not found. retry in 939.980595ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873
21962024-08-23T21:57:06.608ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728") failed: Property not found. retry in 1.241348667s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728
21972024-08-23T21:57:06.701ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb") failed: Property not found. retry in 1.406839186s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb
21982024-08-23T21:57:06.709ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257") failed: Property not found. retry in 1.47995334s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257
21992024-08-23T21:57:06.716ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41") failed: Property not found. retry in 1.187056755s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41
22002024-08-23T21:57:06.725ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473") failed: Property not found. retry in 1.047094537s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473
22012024-08-23T21:57:06.853ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d") failed: Property not found. retry in 902.064963ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d
22022024-08-23T21:57:06.967ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac") failed: Property not found. retry in 1.115184739s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac
22032024-08-23T21:57:07.006ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37") failed: Property not found. retry in 629.455521ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37
22042024-08-23T21:57:07.022ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6") failed: Property not found. retry in 889.005939ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6
22052024-08-23T21:57:07.030ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100") failed: Property not found. retry in 1.259722463s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100
22062024-08-23T21:57:07.217ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18") failed: Property not found. retry in 948.356374ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18
22072024-08-23T21:57:07.316ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320") failed: Property not found. retry in 681.621078ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320
22082024-08-23T21:57:07.353ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873") failed: Property not found. retry in 1.04550559s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873
22092024-08-23T21:57:07.473ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7") failed: Property not found. retry in 1.49729033s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7
22102024-08-23T21:57:07.563ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9") failed: Property not found. retry in 1.023859928s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9
22112024-08-23T21:57:07.581ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6") failed: Property not found. retry in 926.488784ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6
22122024-08-23T21:57:07.589ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e") failed: Property not found. retry in 547.307297ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e
22132024-08-23T21:57:07.644ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37") failed: Property not found. retry in 811.242739ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37
22142024-08-23T21:57:07.766ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d") failed: Property not found. retry in 846.01893ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d
22152024-08-23T21:57:07.782ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473") failed: Property not found. retry in 1.473350252s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473
22162024-08-23T21:57:07.791ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744") failed: Property not found. retry in 1.220043788s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744
22172024-08-23T21:57:07.859ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728") failed: Property not found. retry in 1.127441138s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728
22182024-08-23T21:57:07.913ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41") failed: Property not found. retry in 938.924268ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41
22192024-08-23T21:57:07.921ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6") failed: Property not found. retry in 623.085361ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6
22202024-08-23T21:57:08.008ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320") failed: Property not found. retry in 875.149965ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320
22212024-08-23T21:57:08.091ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac") failed: Property not found. retry in 1.400629979s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac
22222024-08-23T21:57:08.116ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb") failed: Property not found. retry in 1.372714069s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb
22232024-08-23T21:57:08.146ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e") failed: Property not found. retry in 1.027858855s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e
22242024-08-23T21:57:08.185ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18") failed: Property not found. retry in 944.310269ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18
22252024-08-23T21:57:08.206ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257") failed: Property not found. retry in 826.866478ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257
22262024-08-23T21:57:08.303ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100") failed: Property not found. retry in 722.101487ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100
22272024-08-23T21:57:08.411ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873") failed: Property not found. retry in 1.090380512s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873
22282024-08-23T21:57:08.465ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37") failed: Property not found. retry in 1.415451695s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37
22292024-08-23T21:57:08.516ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6") failed: Property not found. retry in 840.078498ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6
22302024-08-23T21:57:08.552ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6") failed: Property not found. retry in 662.785965ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6
22312024-08-23T21:57:08.597ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9") failed: Property not found. retry in 698.338673ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9
22322024-08-23T21:57:08.620ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d") failed: Property not found. retry in 978.217833ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d
22332024-08-23T21:57:08.862ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41") failed: Property not found. retry in 1.003737651s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41
22342024-08-23T21:57:08.892ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320") failed: Property not found. retry in 741.765654ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320
22352024-08-23T21:57:08.978ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7") failed: Property not found. retry in 1.368539003s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7
22362024-08-23T21:57:08.996ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728") failed: Property not found. retry in 847.624711ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728
22372024-08-23T21:57:09.020ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744") failed: Property not found. retry in 966.280388ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744
22382024-08-23T21:57:09.032ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100") failed: Property not found. retry in 869.089993ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100
22392024-08-23T21:57:09.041ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257") failed: Property not found. retry in 715.160053ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257
22402024-08-23T21:57:09.138ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18") failed: Property not found. retry in 1.07362096s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18
22412024-08-23T21:57:09.184ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e") failed: Property not found. retry in 955.83774ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e
22422024-08-23T21:57:09.224ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6") failed: Property not found. retry in 1.292722633s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6
22432024-08-23T21:57:09.265ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473") failed: Property not found. retry in 614.707939ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473
22442024-08-23T21:57:09.303ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9") failed: Property not found. retry in 914.746495ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9
22452024-08-23T21:57:09.365ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6") failed: Property not found. retry in 1.259883858s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6
22462024-08-23T21:57:09.498ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb") failed: Property not found. retry in 1.30824049s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb
22472024-08-23T21:57:09.505ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac") failed: Property not found. retry in 1.39968391s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac
22482024-08-23T21:57:09.511ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873") failed: Property not found. retry in 1.168572595s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873
22492024-08-23T21:57:09.611ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d") failed: Property not found. retry in 1.216447565s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d
22502024-08-23T21:57:09.645ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320") failed: Property not found. retry in 1.142146449s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320
22512024-08-23T21:57:09.768ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257") failed: Property not found. retry in 1.47403258s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257
22522024-08-23T21:57:09.855ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728") failed: Property not found. retry in 858.488971ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728
22532024-08-23T21:57:09.877ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41") failed: Property not found. retry in 1.332654546s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41
22542024-08-23T21:57:09.891ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473") failed: Property not found. retry in 750.843294ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473
22552024-08-23T21:57:09.901ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37") failed: Property not found. retry in 1.152002826s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37
22562024-08-23T21:57:09.912ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100") failed: Property not found. retry in 953.490465ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100
22572024-08-23T21:57:09.915ZINFOSledAgent (ServiceManager): Profile for oxz_clickhouse_8471ffba-0f68-4241-be3b-68581f848fb7: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService21'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::e'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::e'/> <propval type="astring" name="listen_port" value='8123'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="clickhouse_address" value='[fd00:1122:3344:101::e]:8123'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::e]:8888'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
22582024-08-23T21:57:09.918ZINFOSledAgent (ServiceManager): Booting oxz_clickhouse_8471ffba-0f68-4241-be3b-68581f848fb7 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_clickhouse_8471ffba-0f68-4241-be3b-68581f848fb7
22592024-08-23T21:57:09.998ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744") failed: Property not found. retry in 654.881263ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744
22602024-08-23T21:57:10.118ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8471ffba-0f68-4241-be3b-68581f848fb7") failed: Property not found. retry in 31.450823ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8471ffba-0f68-4241-be3b-68581f848fb7
22612024-08-23T21:57:10.153ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e") failed: Property not found. retry in 1.155313829s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e
22622024-08-23T21:57:10.162ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8471ffba-0f68-4241-be3b-68581f848fb7") failed: Property not found. retry in 101.20665ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8471ffba-0f68-4241-be3b-68581f848fb7
22632024-08-23T21:57:10.225ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18") failed: Property not found. retry in 901.9141ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18
22642024-08-23T21:57:10.246ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9") failed: Property not found. retry in 1.477893651s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9
22652024-08-23T21:57:10.278ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8471ffba-0f68-4241-be3b-68581f848fb7") failed: Property not found. retry in 260.356637ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8471ffba-0f68-4241-be3b-68581f848fb7
22662024-08-23T21:57:10.359ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7") failed: Property not found. retry in 1.097013523s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7
22672024-08-23T21:57:10.528ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6") failed: Property not found. retry in 592.405339ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6
22682024-08-23T21:57:10.550ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8471ffba-0f68-4241-be3b-68581f848fb7") failed: Property not found. retry in 211.906706ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8471ffba-0f68-4241-be3b-68581f848fb7
22692024-08-23T21:57:10.638ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6") failed: Property not found. retry in 1.16967083s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6
22702024-08-23T21:57:10.655ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473") failed: Property not found. retry in 1.354577506s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473
22712024-08-23T21:57:10.666ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744") failed: Property not found. retry in 529.9013ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744
22722024-08-23T21:57:10.690ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873") failed: Property not found. retry in 1.149083916s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873
22732024-08-23T21:57:10.725ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728") failed: Property not found. retry in 1.141537332s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728
22742024-08-23T21:57:10.776ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8471ffba-0f68-4241-be3b-68581f848fb7") failed: Property not found. retry in 634.358234ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8471ffba-0f68-4241-be3b-68581f848fb7
22752024-08-23T21:57:10.799ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320") failed: Property not found. retry in 1.463825591s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320
22762024-08-23T21:57:10.818ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb") failed: Property not found. retry in 635.63057ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb
22772024-08-23T21:57:10.839ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d") failed: Property not found. retry in 1.272549518s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d
22782024-08-23T21:57:10.877ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100") failed: Property not found. retry in 626.05677ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100
22792024-08-23T21:57:10.918ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac") failed: Property not found. retry in 839.719219ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac
22802024-08-23T21:57:11.065ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37") failed: Property not found. retry in 1.044106397s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37
22812024-08-23T21:57:11.133ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6") failed: Property not found. retry in 1.419637077s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6
22822024-08-23T21:57:11.144ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18") failed: Property not found. retry in 885.976835ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18
22832024-08-23T21:57:11.206ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744") failed: Property not found. retry in 907.047419ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744
22842024-08-23T21:57:11.222ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41") failed: Property not found. retry in 1.219894574s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41
22852024-08-23T21:57:11.252ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257") failed: Property not found. retry in 629.813136ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257
22862024-08-23T21:57:11.336ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e") failed: Property not found. retry in 1.125543346s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e
22872024-08-23T21:57:11.425ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8471ffba-0f68-4241-be3b-68581f848fb7") failed: Property not found. retry in 1.042870592s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8471ffba-0f68-4241-be3b-68581f848fb7
22882024-08-23T21:57:11.468ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb") failed: Property not found. retry in 918.700498ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb
22892024-08-23T21:57:11.484ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7") failed: Property not found. retry in 1.310463105s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7
22902024-08-23T21:57:11.519ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100") failed: Property not found. retry in 516.920144ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100
22912024-08-23T21:57:11.740ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9") failed: Property not found. retry in 1.202074444s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9
22922024-08-23T21:57:11.772ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac") failed: Property not found. retry in 763.174013ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac
22932024-08-23T21:57:11.822ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6") failed: Property not found. retry in 1.430851417s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6
22942024-08-23T21:57:11.855ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873") failed: Property not found. retry in 1.058396612s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873
22952024-08-23T21:57:11.881ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728") failed: Property not found. retry in 634.161246ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728
22962024-08-23T21:57:11.899ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257") failed: Property not found. retry in 612.869806ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257
22972024-08-23T21:57:12.027ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473") failed: Property not found. retry in 1.105123856s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473
22982024-08-23T21:57:12.056ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18") failed: Property not found. retry in 954.216173ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18
22992024-08-23T21:57:12.086ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100") failed: Property not found. retry in 895.755887ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100
23002024-08-23T21:57:12.124ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37") failed: Property not found. retry in 1.465401479s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37
23012024-08-23T21:57:12.137ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744") failed: Property not found. retry in 927.622417ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744
23022024-08-23T21:57:12.150ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d") failed: Property not found. retry in 1.381906494s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d
23032024-08-23T21:57:12.280ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320") failed: Property not found. retry in 649.175067ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320
23042024-08-23T21:57:12.416ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb") failed: Property not found. retry in 596.004826ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb
23052024-08-23T21:57:12.464ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41") failed: Property not found. retry in 627.568597ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41
23062024-08-23T21:57:12.483ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e") failed: Property not found. retry in 649.056883ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e
23072024-08-23T21:57:12.506ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8471ffba-0f68-4241-be3b-68581f848fb7") failed: Property not found. retry in 1.023895157s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8471ffba-0f68-4241-be3b-68581f848fb7
23082024-08-23T21:57:12.531ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257") failed: Property not found. retry in 784.899047ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257
23092024-08-23T21:57:12.550ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728") failed: Property not found. retry in 1.058953914s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728
23102024-08-23T21:57:12.572ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac") failed: Property not found. retry in 568.210073ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac
23112024-08-23T21:57:12.586ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6") failed: Property not found. retry in 1.491044474s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6
23122024-08-23T21:57:12.810ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7") failed: Property not found. retry in 945.58034ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7
23132024-08-23T21:57:12.934ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873") failed: Property not found. retry in 544.926475ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873
23142024-08-23T21:57:12.954ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320") failed: Property not found. retry in 1.014690537s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320
23152024-08-23T21:57:12.971ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9") failed: Property not found. retry in 1.363389602s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9
23162024-08-23T21:57:12.998ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100") failed: Property not found. retry in 546.561548ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100
23172024-08-23T21:57:13.031ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18") failed: Property not found. retry in 1.044134909s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18
23182024-08-23T21:57:13.054ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb") failed: Property not found. retry in 1.317376056s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb
23192024-08-23T21:57:13.092ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744") failed: Property not found. retry in 555.819445ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744
23202024-08-23T21:57:13.112ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41") failed: Property not found. retry in 1.044535894s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41
23212024-08-23T21:57:13.153ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e") failed: Property not found. retry in 1.300531978s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e
23222024-08-23T21:57:13.181ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473") failed: Property not found. retry in 915.550116ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473
23232024-08-23T21:57:13.224ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac") failed: Property not found. retry in 555.820047ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac
23242024-08-23T21:57:13.268ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6") failed: Property not found. retry in 536.358839ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6
23252024-08-23T21:57:13.330ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257") failed: Property not found. retry in 1.159890804s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257
23262024-08-23T21:57:13.499ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873") failed: Property not found. retry in 1.476792516s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873
23272024-08-23T21:57:13.504ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
23282024-08-23T21:57:13.548ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8471ffba-0f68-4241-be3b-68581f848fb7") failed: Property not found. retry in 1.396234507s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8471ffba-0f68-4241-be3b-68581f848fb7
23292024-08-23T21:57:13.565ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100") failed: Property not found. retry in 954.307119ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100
23302024-08-23T21:57:13.580ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d") failed: Property not found. retry in 555.25042ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d
23312024-08-23T21:57:13.605ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37") failed: Property not found. retry in 827.768122ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37
23322024-08-23T21:57:13.619ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728") failed: Property not found. retry in 770.453517ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728
23332024-08-23T21:57:13.659ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744") failed: Property not found. retry in 1.009927955s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744
23342024-08-23T21:57:13.769ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7") failed: Property not found. retry in 526.469848ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7
23352024-08-23T21:57:13.792ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac") failed: Property not found. retry in 831.452962ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac
23362024-08-23T21:57:13.817ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6") failed: Property not found. retry in 555.54604ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6
23372024-08-23T21:57:13.980ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320") failed: Property not found. retry in 1.477917537s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320
23382024-08-23T21:57:14.088ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18") failed: Property not found. retry in 1.142752412s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18
23392024-08-23T21:57:14.102ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6") failed: Property not found. retry in 1.173369647s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6
23402024-08-23T21:57:14.115ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473") failed: Property not found. retry in 820.67599ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473
23412024-08-23T21:57:14.146ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d") failed: Property not found. retry in 515.30964ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d
23422024-08-23T21:57:14.168ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41") failed: Property not found. retry in 740.049808ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41
23432024-08-23T21:57:14.310ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7") failed: Property not found. retry in 1.373676544s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7
23442024-08-23T21:57:14.350ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9") failed: Property not found. retry in 1.181862141s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9
23452024-08-23T21:57:14.385ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb") failed: Property not found. retry in 678.656607ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb
23462024-08-23T21:57:14.396ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6") failed: Property not found. retry in 679.601013ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6
23472024-08-23T21:57:14.407ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728") failed: Property not found. retry in 785.638062ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728
23482024-08-23T21:57:14.446ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37") failed: Property not found. retry in 711.631657ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37
23492024-08-23T21:57:14.468ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e") failed: Property not found. retry in 1.272544717s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e
23502024-08-23T21:57:14.504ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257") failed: Property not found. retry in 1.037496812s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257
23512024-08-23T21:57:14.532ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100") failed: Property not found. retry in 1.139516421s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100
23522024-08-23T21:57:14.638ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac") failed: Property not found. retry in 696.779283ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac
23532024-08-23T21:57:14.674ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d") failed: Property not found. retry in 730.261572ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d
23542024-08-23T21:57:14.689ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744") failed: Property not found. retry in 875.001688ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744
23552024-08-23T21:57:14.924ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41") failed: Property not found. retry in 1.366777599s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41
23562024-08-23T21:57:14.950ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473") failed: Property not found. retry in 1.086158906s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473
23572024-08-23T21:57:14.963ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8471ffba-0f68-4241-be3b-68581f848fb7") failed: Property not found. retry in 1.180023032s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8471ffba-0f68-4241-be3b-68581f848fb7
23582024-08-23T21:57:14.992ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873") failed: Property not found. retry in 1.143439667s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873
23592024-08-23T21:57:15.079ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb") failed: Property not found. retry in 1.476078179s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb
23602024-08-23T21:57:15.096ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6") failed: Property not found. retry in 1.460489041s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6
23612024-08-23T21:57:15.175ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37") failed: Property not found. retry in 919.850576ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37
23622024-08-23T21:57:15.209ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728") failed: Property not found. retry in 580.207551ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728
23632024-08-23T21:57:15.248ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18") failed: Property not found. retry in 1.344909817s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18
23642024-08-23T21:57:15.296ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6") failed: Property not found. retry in 1.309868764s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6
23652024-08-23T21:57:15.353ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac") failed: Property not found. retry in 1.096876291s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac
23662024-08-23T21:57:15.420ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d") failed: Property not found. retry in 1.478290713s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d
23672024-08-23T21:57:15.476ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320") failed: Property not found. retry in 1.224395722s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320
23682024-08-23T21:57:15.545ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9") failed: Property not found. retry in 637.898802ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9
23692024-08-23T21:57:15.561ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257") failed: Property not found. retry in 880.484932ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257
23702024-08-23T21:57:15.580ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744") failed: Property not found. retry in 914.899163ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744
23712024-08-23T21:57:15.689ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100") failed: Property not found. retry in 1.184350711s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100
23722024-08-23T21:57:15.703ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7") failed: Property not found. retry in 906.894293ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7
23732024-08-23T21:57:15.755ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e") failed: Property not found. retry in 956.368569ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e
23742024-08-23T21:57:15.802ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728") failed: Property not found. retry in 1.135391106s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728
23752024-08-23T21:57:16.058ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473") failed: Property not found. retry in 1.230848759s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473
23762024-08-23T21:57:16.113ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37") failed: Property not found. retry in 1.333252589s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37
23772024-08-23T21:57:16.150ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873") failed: Property not found. retry in 701.773927ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873
23782024-08-23T21:57:16.173ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8471ffba-0f68-4241-be3b-68581f848fb7") failed: Property not found. retry in 1.471430387s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8471ffba-0f68-4241-be3b-68581f848fb7
23792024-08-23T21:57:16.203ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9") failed: Property not found. retry in 1.10262019s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9
23802024-08-23T21:57:16.311ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41") failed: Property not found. retry in 1.356646703s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41
23812024-08-23T21:57:16.456ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257") failed: Property not found. retry in 1.131921273s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257
23822024-08-23T21:57:16.469ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac") failed: Property not found. retry in 1.345945191s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac
23832024-08-23T21:57:16.515ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744") failed: Property not found. retry in 729.719508ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744
23842024-08-23T21:57:16.576ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb") failed: Property not found. retry in 743.303972ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb
23852024-08-23T21:57:16.590ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6") failed: Property not found. retry in 1.106961584s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6
23862024-08-23T21:57:16.610ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18") failed: Property not found. retry in 521.378468ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18
23872024-08-23T21:57:16.627ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7") failed: Property not found. retry in 1.480565814s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7
23882024-08-23T21:57:16.640ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6") failed: Property not found. retry in 950.72885ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6
23892024-08-23T21:57:16.716ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320") failed: Property not found. retry in 767.766907ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320
23902024-08-23T21:57:16.732ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e") failed: Property not found. retry in 529.367213ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e
23912024-08-23T21:57:16.867ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873") failed: Property not found. retry in 733.765181ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873
23922024-08-23T21:57:16.890ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100") failed: Property not found. retry in 1.040874403s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100
23932024-08-23T21:57:16.913ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d") failed: Property not found. retry in 1.47093322s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d
23942024-08-23T21:57:16.949ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728") failed: Property not found. retry in 647.027405ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728
23952024-08-23T21:57:17.144ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18") failed: Property not found. retry in 685.795028ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18
23962024-08-23T21:57:17.259ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744") failed: Property not found. retry in 714.78632ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744
23972024-08-23T21:57:17.275ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e") failed: Property not found. retry in 925.942036ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e
23982024-08-23T21:57:17.303ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473") failed: Property not found. retry in 532.042951ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473
23992024-08-23T21:57:17.318ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9") failed: Property not found. retry in 772.562436ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9
24002024-08-23T21:57:17.332ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb") failed: Property not found. retry in 1.030695229s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb
24012024-08-23T21:57:17.461ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37") failed: Property not found. retry in 1.228436544s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37
24022024-08-23T21:57:17.497ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320") failed: Property not found. retry in 563.437307ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320
24032024-08-23T21:57:17.602ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257") failed: Property not found. retry in 1.165135606s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257
24042024-08-23T21:57:17.613ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6") failed: Property not found. retry in 1.042213187s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6
24052024-08-23T21:57:17.624ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728") failed: Property not found. retry in 1.032474021s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728
24062024-08-23T21:57:17.665ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8471ffba-0f68-4241-be3b-68581f848fb7") failed: Property not found. retry in 1.280199521s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8471ffba-0f68-4241-be3b-68581f848fb7
24072024-08-23T21:57:17.690ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41") failed: Property not found. retry in 535.029713ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41
24082024-08-23T21:57:17.720ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6") failed: Property not found. retry in 736.651326ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6
24092024-08-23T21:57:17.847ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18") failed: Property not found. retry in 902.970702ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18
24102024-08-23T21:57:18.079ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320") failed: Property not found. retry in 871.27401ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320
24112024-08-23T21:57:18.134ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7") failed: Property not found. retry in 932.81124ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7
24122024-08-23T21:57:19.042ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8471ffba-0f68-4241-be3b-68581f848fb7") failed: Property not found. retry in 853.469368ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8471ffba-0f68-4241-be3b-68581f848fb7
24132024-08-23T21:57:19.951ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3245 zone = oxz_crucible_914d3516-8bfd-4006-87d8-91481730f9c7
24142024-08-23T21:57:19.951ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3245 zone = oxz_external_dns_1db57e91-9837-462c-9f2c-1b4fca780b7e
24152024-08-23T21:57:19.951ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3245 zone = oxz_crucible_92744278-0a6d-41bc-a502-b694867741c6
24162024-08-23T21:57:19.951ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3245 zone = oxz_crucible_pantry_1bb815f5-1d70-43b2-a7b7-1bb132d87fd6
24172024-08-23T21:57:19.951ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3245 zone = oxz_nexus_f9a2f81c-00cf-49a5-a96e-1c4dfeba1728
24182024-08-23T21:57:19.951ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3245 zone = oxz_oximeter_ba8b1177-03a3-49b6-a6ee-b364652c50c9
24192024-08-23T21:57:19.951ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3245 zone = oxz_crucible_3dafd3b1-1230-4db3-9961-2a4d53cc3a37
24202024-08-23T21:57:19.951ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3245 zone = oxz_crucible_pantry_d9421cb5-f5bb-43e4-a2f0-337cff4f1100
24212024-08-23T21:57:19.951ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3245 zone = oxz_external_dns_b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac
24222024-08-23T21:57:19.951ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3245 zone = oxz_nexus_2c94012a-643d-4f61-8185-c8786e00e320
24232024-08-23T21:57:19.951ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3245 zone = oxz_crucible_66dd6deb-9c39-47ef-9f1c-98fea2467257
24242024-08-23T21:57:19.951ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3245 zone = oxz_crucible_9c2bbea7-8b8c-4d83-9618-26c0848936eb
24252024-08-23T21:57:19.951ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3245 zone = oxz_nexus_4e26237f-8cdc-4da1-b44a-9b00d20bd873
24262024-08-23T21:57:19.951ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3245 zone = oxz_crucible_2bda41aa-8135-413a-853b-44e7fc746744
24272024-08-23T21:57:19.951ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3245 zone = oxz_crucible_581275cf-e6a7-4ef5-ad1e-36b1379b2a18
24282024-08-23T21:57:19.951ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3245 zone = oxz_crucible_pantry_8343e6aa-08fb-4b4a-bf94-a818d66d575d
24292024-08-23T21:57:19.951ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3245 zone = oxz_clickhouse_8471ffba-0f68-4241-be3b-68581f848fb7
24302024-08-23T21:57:19.951ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3245 zone = oxz_crucible_83653856-00c4-44fe-b4df-ad99e78af473
24312024-08-23T21:57:19.951ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3245 zone = oxz_crucible_02433f6a-f8a9-4f05-af1b-be59dfd00a41
24322024-08-23T21:57:19.952ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
24332024-08-23T21:57:19.956ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
24342024-08-23T21:57:19.959ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 38222368 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:55451 req_id = c3022fca-c201-49c6-86ba-ebdc97feba00 response_code = 204 uri = /omicron-zones
24352024-08-23T21:57:19.988ZINFOSledAgent (RSS): Finished setting up services file = sled-agent/src/rack_setup/service.rs:1286
24362024-08-23T21:57:19.989ZINFOSledAgent (RSS): Writing ledger to /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
24372024-08-23T21:57:19.989ZINFOSledAgent (RSS): Writing ledger to /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
24382024-08-23T21:57:19.993ZINFOSledAgent (RSS): Handing off control to Nexus file = sled-agent/src/rack_setup/service.rs:689
24392024-08-23T21:57:19.999ZINFOSledAgent (RSS): Nexus address: [fd00:1122:3344:101::b]:12221 file = sled-agent/src/rack_setup/service.rs:702
24402024-08-23T21:57:20.065ZINFOSledAgent (RSS): rack_network_config: RackNetworkConfigV2 { bfd: [], bgp: [], infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV2 { addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.1.174, width: 24, }, ), vlan_id: None, }, ], autoneg: false, bgp_peers: [], lldp: None, port: "qsfp0", routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), local_pref: None, nexthop: 10.151.1.100, vlan_id: None, }, ], switch: Switch0, uplink_port_fec: None, uplink_port_speed: Speed40G, }, ], rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, } file = sled-agent/src/rack_setup/service.rs:867
24412024-08-23T21:57:20.068ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/da7dff3e-1935-4d21-ab58-793c0302e3ea/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
24422024-08-23T21:57:20.216ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/da7dff3e-1935-4d21-ab58-793c0302e3ea/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
24432024-08-23T21:57:20.282ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/da7dff3e-1935-4d21-ab58-793c0302e3ea/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
24442024-08-23T21:57:20.443ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/da7dff3e-1935-4d21-ab58-793c0302e3ea/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
24452024-08-23T21:57:20.543ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/da7dff3e-1935-4d21-ab58-793c0302e3ea/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
24462024-08-23T21:57:20.698ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/da7dff3e-1935-4d21-ab58-793c0302e3ea/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
24472024-08-23T21:57:20.878ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/da7dff3e-1935-4d21-ab58-793c0302e3ea/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
24482024-08-23T21:57:21.105ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/da7dff3e-1935-4d21-ab58-793c0302e3ea/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
24492024-08-23T21:57:21.403ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/da7dff3e-1935-4d21-ab58-793c0302e3ea/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
24502024-08-23T21:57:21.624ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/da7dff3e-1935-4d21-ab58-793c0302e3ea/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
24512024-08-23T21:57:22.279ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/da7dff3e-1935-4d21-ab58-793c0302e3ea/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
24522024-08-23T21:57:23.094ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/da7dff3e-1935-4d21-ab58-793c0302e3ea/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
24532024-08-23T21:57:23.771ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/da7dff3e-1935-4d21-ab58-793c0302e3ea/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
24542024-08-23T21:57:24.556ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/da7dff3e-1935-4d21-ab58-793c0302e3ea/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
24552024-08-23T21:57:25.940ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/da7dff3e-1935-4d21-ab58-793c0302e3ea/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
24562024-08-23T21:57:26.048ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:61363
24572024-08-23T21:57:26.049ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 161 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:61363 req_id = 298145b0-dd78-4b8c-8df6-90688a668584 response_code = 200 uri = /baseboard
24582024-08-23T21:57:27.060ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/da7dff3e-1935-4d21-ab58-793c0302e3ea/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
24592024-08-23T21:57:28.549ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/da7dff3e-1935-4d21-ab58-793c0302e3ea/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
24602024-08-23T21:57:30.648ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "76f06a97-d058-4f33-8f27-5c20bea0c352", "content-length": "173", "date": "Fri, 23 Aug 2024 21:57:30 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: rack with id \"da7dff3e-1935-4d21-ab58-793c0302e3ea\"", request_id: "76f06a97-d058-4f33-8f27-5c20bea0c352" } file = sled-agent/src/rack_setup/service.rs:929
24612024-08-23T21:57:34.309ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "e2162715-de37-4bcd-bef2-9225dd65baa9", "content-length": "173", "date": "Fri, 23 Aug 2024 21:57:33 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"5050fe14-0845-40b2-872d-d70a5b2b56be\"", request_id: "e2162715-de37-4bcd-bef2-9225dd65baa9" } file = sled-agent/src/rack_setup/service.rs:929
24622024-08-23T21:57:37.744ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "3f1757ef-0a45-4294-88df-e1bdd1c3add9", "content-length": "173", "date": "Fri, 23 Aug 2024 21:57:37 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"5050fe14-0845-40b2-872d-d70a5b2b56be\"", request_id: "3f1757ef-0a45-4294-88df-e1bdd1c3add9" } file = sled-agent/src/rack_setup/service.rs:929
24632024-08-23T21:57:42.357ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "42d65ef7-b71e-4c98-80b9-c89d7f055cd2", "content-length": "173", "date": "Fri, 23 Aug 2024 21:57:42 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"5050fe14-0845-40b2-872d-d70a5b2b56be\"", request_id: "42d65ef7-b71e-4c98-80b9-c89d7f055cd2" } file = sled-agent/src/rack_setup/service.rs:929
24642024-08-23T21:57:45.752ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "8a0834e9-c1e2-4559-8232-2a8eeae3e80e", "content-length": "173", "date": "Fri, 23 Aug 2024 21:57:45 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"5050fe14-0845-40b2-872d-d70a5b2b56be\"", request_id: "8a0834e9-c1e2-4559-8232-2a8eeae3e80e" } file = sled-agent/src/rack_setup/service.rs:929
24652024-08-23T21:57:51.460ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "0f68f113-2468-4510-a297-f321fb8c1053", "content-length": "173", "date": "Fri, 23 Aug 2024 21:57:51 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"5050fe14-0845-40b2-872d-d70a5b2b56be\"", request_id: "0f68f113-2468-4510-a297-f321fb8c1053" } file = sled-agent/src/rack_setup/service.rs:929
24662024-08-23T21:57:54.998ZINFOSledAgent (NexusNotifierTask): Successfully put SledAgentInfo to nexus file = sled-agent/src/nexus.rs:544 new_generation = 1 old_generation = 1 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
24672024-08-23T21:57:55.139ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47633
24682024-08-23T21:57:55.142ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
24692024-08-23T21:57:55.232ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 92339 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47633 req_id = 10475df8-de1f-48a6-8d0e-9674da7d7605 response_code = 200 uri = /inventory
24702024-08-23T21:57:55.234ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
24712024-08-23T21:57:55.234ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
24722024-08-23T21:57:55.241ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
24732024-08-23T21:57:55.241ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 9357 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47633 req_id = 83dad523-2dbb-476b-a0ec-442f8a4b4296 response_code = 200 uri = /omicron-zones
24742024-08-23T21:57:56.084ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:39724
24752024-08-23T21:57:56.084ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 37 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:39724 req_id = dab9ed0d-0968-4946-8fcf-e837b9c9f2d2 response_code = 200 uri = /baseboard
24762024-08-23T21:58:00.477ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37532
24772024-08-23T21:58:00.482ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 2a9b2cf7-2fb3-4597-871e-44332b4d82f2, kind: Service { id: 1db57e91-9837-462c-9f2c-1b4fca780b7e }, name: Name("external-dns-1db57e91-9837-462c-9f2c-1b4fca780b7e"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 231, 24])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 5d1239c9-696e-4616-a2a6-82968e91a2e1, kind: Service { id: b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac }, name: Name("external-dns-b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 184])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 00000426-1f68-402a-a60e-4fa2f73c115f, kind: Service { id: 4e26237f-8cdc-4da1-b44a-9b00d20bd873 }, name: Name("nexus-4e26237f-8cdc-4da1-b44a-9b00d20bd873"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 190, 168])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 5cf061ea-987b-4986-8378-702c868c90c5, kind: Service { id: f9a2f81c-00cf-49a5-a96e-1c4dfeba1728 }, name: Name("nexus-f9a2f81c-00cf-49a5-a96e-1c4dfeba1728"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 196, 138])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: b2284d48-b022-4af4-a2f6-3a691122eb62, kind: Service { id: 2c94012a-643d-4f61-8185-c8786e00e320 }, name: Name("nexus-2c94012a-643d-4f61-8185-c8786e00e320"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 253, 42])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
24782024-08-23T21:58:00.482ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
24792024-08-23T21:58:00.482ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
24802024-08-23T21:58:00.482ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
24812024-08-23T21:58:00.482ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
24822024-08-23T21:58:00.482ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
24832024-08-23T21:58:00.482ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
24842024-08-23T21:58:00.482ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 3550 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:37532 req_id = f85bba36-4bbf-4c89-b667-b4b988b8b079 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
24852024-08-23T21:58:00.484ZINFOSledAgent (RSS): Handoff to Nexus is complete file = sled-agent/src/rack_setup/service.rs:939
24862024-08-23T21:58:00.733ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:51725
24872024-08-23T21:58:00.734ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1105 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:51725 req_id = 21ae1afc-4005-454c-8cea-67697206034a response_code = 200 uri = /vpc-routes
24882024-08-23T21:58:00.742ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:51045
24892024-08-23T21:58:00.742ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
24902024-08-23T21:58:00.836ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 94343 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:51045 req_id = 641f5f76-b3bf-4dbc-9a02-7d460541d514 response_code = 200 uri = /inventory
24912024-08-23T21:58:00.837ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:46956
24922024-08-23T21:58:00.837ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63190
24932024-08-23T21:58:00.838ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
24942024-08-23T21:58:00.838ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/config/omicron-zones.json file = common/src/ledger.rs:177
24952024-08-23T21:58:00.838ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 2a9b2cf7-2fb3-4597-871e-44332b4d82f2, kind: Service { id: 1db57e91-9837-462c-9f2c-1b4fca780b7e }, name: Name("external-dns-1db57e91-9837-462c-9f2c-1b4fca780b7e"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 231, 24])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 5d1239c9-696e-4616-a2a6-82968e91a2e1, kind: Service { id: b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac }, name: Name("external-dns-b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 184])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 00000426-1f68-402a-a60e-4fa2f73c115f, kind: Service { id: 4e26237f-8cdc-4da1-b44a-9b00d20bd873 }, name: Name("nexus-4e26237f-8cdc-4da1-b44a-9b00d20bd873"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 190, 168])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 5cf061ea-987b-4986-8378-702c868c90c5, kind: Service { id: f9a2f81c-00cf-49a5-a96e-1c4dfeba1728 }, name: Name("nexus-f9a2f81c-00cf-49a5-a96e-1c4dfeba1728"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 196, 138])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: b2284d48-b022-4af4-a2f6-3a691122eb62, kind: Service { id: 2c94012a-643d-4f61-8185-c8786e00e320 }, name: Name("nexus-2c94012a-643d-4f61-8185-c8786e00e320"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 253, 42])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
24962024-08-23T21:58:00.838ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 2a9b2cf7-2fb3-4597-871e-44332b4d82f2, kind: Service { id: 1db57e91-9837-462c-9f2c-1b4fca780b7e }, name: Name("external-dns-1db57e91-9837-462c-9f2c-1b4fca780b7e"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 231, 24])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 5d1239c9-696e-4616-a2a6-82968e91a2e1, kind: Service { id: b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac }, name: Name("external-dns-b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 184])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 00000426-1f68-402a-a60e-4fa2f73c115f, kind: Service { id: 4e26237f-8cdc-4da1-b44a-9b00d20bd873 }, name: Name("nexus-4e26237f-8cdc-4da1-b44a-9b00d20bd873"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 190, 168])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 5cf061ea-987b-4986-8378-702c868c90c5, kind: Service { id: f9a2f81c-00cf-49a5-a96e-1c4dfeba1728 }, name: Name("nexus-f9a2f81c-00cf-49a5-a96e-1c4dfeba1728"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 196, 138])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: b2284d48-b022-4af4-a2f6-3a691122eb62, kind: Service { id: 2c94012a-643d-4f61-8185-c8786e00e320 }, name: Name("nexus-2c94012a-643d-4f61-8185-c8786e00e320"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 253, 42])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
24972024-08-23T21:58:00.838ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
24982024-08-23T21:58:00.838ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
24992024-08-23T21:58:00.838ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
25002024-08-23T21:58:00.838ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
25012024-08-23T21:58:00.838ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
25022024-08-23T21:58:00.838ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/config/omicron-zones.json file = common/src/ledger.rs:177
25032024-08-23T21:58:00.838ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
25042024-08-23T21:58:00.838ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1455 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:63190 req_id = 21f4be17-ddff-4b82-abe8-644b326f891b response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
25052024-08-23T21:58:00.838ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
25062024-08-23T21:58:00.838ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
25072024-08-23T21:58:00.838ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
25082024-08-23T21:58:00.838ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
25092024-08-23T21:58:00.838ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
25102024-08-23T21:58:00.838ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
25112024-08-23T21:58:00.838ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1394 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:46956 req_id = fc3109e4-d332-4bea-b954-174bfe423dd9 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
25122024-08-23T21:58:00.839ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1772 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:51045 req_id = 95300ecc-594b-456d-a904-d4b236735a54 response_code = 200 uri = /omicron-zones
25132024-08-23T21:58:00.840ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 3175 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:51725 req_id = d2c1aeb0-4e63-40ee-8f60-ecdb8907d331 response_code = 204 uri = /vpc-routes
25142024-08-23T21:58:00.893ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63332
25152024-08-23T21:58:00.894ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 637 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:63332 req_id = 0667731b-dd7b-4697-8dbb-4f4e87dc4594 response_code = 200 uri = /v2p
25162024-08-23T21:58:01.159ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:47819
25172024-08-23T21:58:01.160ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
25182024-08-23T21:58:01.276ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
25192024-08-23T21:58:01.358ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 199570 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:47819 req_id = 124fec58-2444-4214-8086-7f995e17f017 response_code = 204 uri = /switch-ports
25202024-08-23T21:58:01.359ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63432
25212024-08-23T21:58:01.360ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1158 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:63432 req_id = 1c942b21-f2c7-4f4e-a38d-971d866990df response_code = 200 uri = /vpc-routes
25222024-08-23T21:58:01.367ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8654 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:47819 req_id = f426ad22-5e70-4de1-bd66-4ef87277a3a8 response_code = 200 uri = /network-bootstore-config
25232024-08-23T21:58:01.369ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 184 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:63432 req_id = eaf82a21-189c-40b6-a587-60f596a5e333 response_code = 204 uri = /vpc-routes
25242024-08-23T21:58:01.405ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:46720
25252024-08-23T21:58:01.405ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 117 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:46720 req_id = f77e062d-6763-4466-aca6-8563b2703520 response_code = 200 uri = /v2p
25262024-08-23T21:58:01.429ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:56988
25272024-08-23T21:58:01.430ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1090 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:56988 req_id = cc9e4e5f-8a59-4960-8d52-d0e1bf068ad9 response_code = 200 uri = /vpc-routes
25282024-08-23T21:58:01.441ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 80 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:56988 req_id = a1cdb777-01e7-473b-ba2a-274a1b9a62ec response_code = 204 uri = /vpc-routes
25292024-08-23T21:58:01.506ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42741
25302024-08-23T21:58:01.506ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 146 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:42741 req_id = c243ebbd-d0b2-460a-bf34-f32527e9fc5a response_code = 200 uri = /v2p
25312024-08-23T21:58:01.506ZINFOSledAgent (bootstore): Attempting to update network config with generation: 2, current_generation: 1 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-buskin-i86pc
25322024-08-23T21:58:01.506ZINFOSledAgent (bootstore): Writing ledger to /pool/int/156808b0-3956-455e-a8e1-2217e0afca54/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
25332024-08-23T21:58:01.508ZINFOSledAgent (bootstore): Writing ledger to /pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
25342024-08-23T21:58:01.510ZINFOSledAgent (bootstore): Broadcasting network config with generation 2 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-buskin-i86pc
25352024-08-23T21:58:01.510ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 5954 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:47819 req_id = 43fd1620-cc2d-4445-8787-e54ffc916e51 response_code = 204 uri = /network-bootstore-config
25362024-08-23T21:58:01.579ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44182
25372024-08-23T21:58:01.579ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
25382024-08-23T21:58:01.621ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
25392024-08-23T21:58:01.684ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 104520 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:44182 req_id = b764661d-827b-4442-9f2d-1d6afafb9744 response_code = 204 uri = /switch-ports
25402024-08-23T21:58:01.684ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:38526
25412024-08-23T21:58:01.684ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 146 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44182 req_id = aacbc3a5-10dd-4c82-8aa0-df66ec9a2507 response_code = 200 uri = /network-bootstore-config
25422024-08-23T21:58:01.685ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
25432024-08-23T21:58:01.725ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
25442024-08-23T21:58:01.785ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 101010 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:38526 req_id = 0c44e61b-f089-4709-977b-1d81a93fa9bf response_code = 204 uri = /switch-ports
25452024-08-23T21:58:01.786ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 173 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:38526 req_id = fc0204bc-d4e0-47fe-ba15-92116ff79be4 response_code = 200 uri = /network-bootstore-config
25462024-08-23T21:58:02.066ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:58328
25472024-08-23T21:58:02.066ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 2a9b2cf7-2fb3-4597-871e-44332b4d82f2, kind: Service { id: 1db57e91-9837-462c-9f2c-1b4fca780b7e }, name: Name("external-dns-1db57e91-9837-462c-9f2c-1b4fca780b7e"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 231, 24])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 5d1239c9-696e-4616-a2a6-82968e91a2e1, kind: Service { id: b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac }, name: Name("external-dns-b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 184])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 00000426-1f68-402a-a60e-4fa2f73c115f, kind: Service { id: 4e26237f-8cdc-4da1-b44a-9b00d20bd873 }, name: Name("nexus-4e26237f-8cdc-4da1-b44a-9b00d20bd873"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 190, 168])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 5cf061ea-987b-4986-8378-702c868c90c5, kind: Service { id: f9a2f81c-00cf-49a5-a96e-1c4dfeba1728 }, name: Name("nexus-f9a2f81c-00cf-49a5-a96e-1c4dfeba1728"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 196, 138])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: b2284d48-b022-4af4-a2f6-3a691122eb62, kind: Service { id: 2c94012a-643d-4f61-8185-c8786e00e320 }, name: Name("nexus-2c94012a-643d-4f61-8185-c8786e00e320"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 253, 42])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
25482024-08-23T21:58:02.066ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
25492024-08-23T21:58:02.066ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
25502024-08-23T21:58:02.066ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
25512024-08-23T21:58:02.066ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
25522024-08-23T21:58:02.066ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
25532024-08-23T21:58:02.066ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
25542024-08-23T21:58:02.066ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 407 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:58328 req_id = 8053885b-d808-4e96-aee3-328aef377369 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
25552024-08-23T21:58:08.452ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:41348
25562024-08-23T21:58:08.452ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 49 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:41348 req_id = cb85813b-0737-437f-b565-cb79e7ca1a9e response_code = 200 uri = /vpc-routes
25572024-08-23T21:58:08.459ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 96 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:41348 req_id = 08f10cbd-04e6-4c11-b721-dbd8462bf7ef response_code = 204 uri = /vpc-routes
25582024-08-23T21:58:09.218ZINFOSledAgent (dropshot): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:46625 remote_addr = [fd00:1122:3344:101::d]:62985 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
25592024-08-23T21:58:09.282ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 63639 local_addr = [fd00:1122:3344:101::1]:46625 method = GET remote_addr = [fd00:1122:3344:101::d]:62985 req_id = 505040f2-40fe-46fe-91a3-7d9fb5fa1f40 response_code = 200 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be uri = /5050fe14-0845-40b2-872d-d70a5b2b56be
25602024-08-23T21:58:26.122ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:44519
25612024-08-23T21:58:26.123ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 37 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:44519 req_id = 20c5d00f-1c66-4594-9c45-afbe569492ec response_code = 200 uri = /baseboard
25622024-08-23T21:58:30.659ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:62129
25632024-08-23T21:58:30.659ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 79 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:62129 req_id = 289585a5-fcab-40f4-975a-534b4799ca47 response_code = 200 uri = /vpc-routes
25642024-08-23T21:58:30.779ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 112 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:62129 req_id = b1d9b731-11e2-4ebb-af15-f1767fbf92cc response_code = 204 uri = /vpc-routes
25652024-08-23T21:58:30.841ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63637
25662024-08-23T21:58:30.841ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 109 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:63637 req_id = 54edfc61-89af-4abd-9d18-0fb18e6419a9 response_code = 200 uri = /v2p
25672024-08-23T21:58:31.289ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:62158
25682024-08-23T21:58:31.289ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
25692024-08-23T21:58:31.333ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
25702024-08-23T21:58:31.395ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 106024 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:62158 req_id = 60f10eec-f412-471f-9473-5c295127ecc4 response_code = 204 uri = /switch-ports
25712024-08-23T21:58:31.397ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1287 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:62158 req_id = acd237aa-6855-4c75-bf6e-9572cb68ceaa response_code = 200 uri = /network-bootstore-config
25722024-08-23T21:58:31.450ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:54205
25732024-08-23T21:58:31.450ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 170 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:54205 req_id = e564e7bd-b2b5-4806-acde-760516d0d7fc response_code = 200 uri = /v2p
25742024-08-23T21:58:31.531ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:60128
25752024-08-23T21:58:31.531ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 64 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:60128 req_id = 6d1a9e50-f2f0-4c12-a04f-898ddb1e3527 response_code = 200 uri = /vpc-routes
25762024-08-23T21:58:31.541ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1148 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:60128 req_id = db3db577-9393-42b0-9cb5-c5e5bef76283 response_code = 204 uri = /vpc-routes
25772024-08-23T21:58:31.613ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:41002
25782024-08-23T21:58:31.613ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 47 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:41002 req_id = 3fb6f394-4411-4cba-8e0e-b77e23452cba response_code = 200 uri = /vpc-routes
25792024-08-23T21:58:31.622ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 85 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:41002 req_id = 31f7b90d-b3a9-4e0c-aaf0-4f7e0d97d670 response_code = 204 uri = /vpc-routes
25802024-08-23T21:58:31.628ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:45586
25812024-08-23T21:58:31.628ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:45586 req_id = 728f5502-a903-42d8-b1b0-52129a599f36 response_code = 200 uri = /v2p
25822024-08-23T21:58:31.786ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:52537
25832024-08-23T21:58:31.786ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
25842024-08-23T21:58:31.827ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
25852024-08-23T21:58:31.883ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 97480 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:52537 req_id = 8bd77a2f-4325-4aa0-b324-d01968f88a46 response_code = 204 uri = /switch-ports
25862024-08-23T21:58:31.883ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34662
25872024-08-23T21:58:31.884ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
25882024-08-23T21:58:31.887ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 3259 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:52537 req_id = 0dfb381b-bb07-4d7c-9eab-09e4d513bca2 response_code = 200 uri = /network-bootstore-config
25892024-08-23T21:58:31.917ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
25902024-08-23T21:58:31.976ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 92988 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:34662 req_id = f0bcb50b-3fac-4a60-ba5d-1b218cd589e1 response_code = 204 uri = /switch-ports
25912024-08-23T21:58:31.977ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 162 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:34662 req_id = 4e9bffd7-fb06-41a0-9dfa-fabe28a8bbfc response_code = 200 uri = /network-bootstore-config
25922024-08-23T21:58:39.227ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8260 local_addr = [fd00:1122:3344:101::1]:46625 method = GET remote_addr = [fd00:1122:3344:101::d]:62985 req_id = 072e7ee1-b3a8-430f-8d49-2805f6f88d84 response_code = 200 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be uri = /5050fe14-0845-40b2-872d-d70a5b2b56be
25932024-08-23T21:58:56.159ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:34215
25942024-08-23T21:58:56.159ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 64 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:34215 req_id = 2022e32a-2be4-4240-adf6-c31137d9d665 response_code = 200 uri = /baseboard
25952024-08-23T21:59:00.711ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:64481
25962024-08-23T21:59:00.711ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 101 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:64481 req_id = 82260bf6-2b45-4502-892e-6ad8dfbd3812 response_code = 200 uri = /vpc-routes
25972024-08-23T21:59:00.808ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 151 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:64481 req_id = 28af3017-a774-411c-8e6a-f6d4f5e882c2 response_code = 204 uri = /vpc-routes
25982024-08-23T21:59:00.889ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:40714
25992024-08-23T21:59:00.889ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 133 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:40714 req_id = 85b11506-22fb-4a53-9f86-5af68560dd2d response_code = 200 uri = /v2p
26002024-08-23T21:59:01.103ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:58029
26012024-08-23T21:59:01.104ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
26022024-08-23T21:59:01.153ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
26032024-08-23T21:59:01.229ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 125551 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:58029 req_id = 9e9726e7-b043-4019-b107-c78a3de8160e response_code = 204 uri = /switch-ports
26042024-08-23T21:59:01.230ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 769 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:58029 req_id = debe5013-57ba-4ab2-ba09-c7cecd03009c response_code = 200 uri = /network-bootstore-config
26052024-08-23T21:59:01.252ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:56431
26062024-08-23T21:59:01.253ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 928 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:56431 req_id = ace96494-5527-4468-ad52-943f14cebca9 response_code = 200 uri = /vpc-routes
26072024-08-23T21:59:01.259ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 118 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:56431 req_id = 36412510-6696-48cf-9d74-bcbfcb7414c7 response_code = 204 uri = /vpc-routes
26082024-08-23T21:59:01.297ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:33328
26092024-08-23T21:59:01.297ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 113 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:33328 req_id = 81969f7d-641e-4489-bd85-71b4498b65da response_code = 200 uri = /v2p
26102024-08-23T21:59:01.419ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58064
26112024-08-23T21:59:01.419ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
26122024-08-23T21:59:01.462ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
26132024-08-23T21:59:01.517ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 98200 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:58064 req_id = 16ae5765-f887-4c08-ad04-6e46ada845fd response_code = 204 uri = /switch-ports
26142024-08-23T21:59:01.518ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57181
26152024-08-23T21:59:01.518ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41720
26162024-08-23T21:59:01.518ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 105 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:57181 req_id = f37b072b-3de8-46ef-8270-703f816865de response_code = 200 uri = /vpc-routes
26172024-08-23T21:59:01.518ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 270 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58064 req_id = dc875cb3-88c5-4c64-bcbf-cc2e324bf9fd response_code = 200 uri = /network-bootstore-config
26182024-08-23T21:59:01.518ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 222 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41720 req_id = 56b74022-28da-47a5-b374-5e99f2d292e0 response_code = 200 uri = /v2p
26192024-08-23T21:59:01.525ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 662 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:57181 req_id = a4c87957-15e0-47f2-848e-3e6aa45cf419 response_code = 204 uri = /vpc-routes
26202024-08-23T21:59:01.553ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:49617
26212024-08-23T21:59:01.553ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
26222024-08-23T21:59:01.591ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
26232024-08-23T21:59:01.645ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 92008 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:49617 req_id = d1847014-ddbf-4030-bf57-3acdac229f9a response_code = 204 uri = /switch-ports
26242024-08-23T21:59:01.646ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 601 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:49617 req_id = 48127ef7-5b65-40ce-8a4d-c4bc6eb81035 response_code = 200 uri = /network-bootstore-config
26252024-08-23T21:59:09.224ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 5676 local_addr = [fd00:1122:3344:101::1]:46625 method = GET remote_addr = [fd00:1122:3344:101::d]:62985 req_id = f0585996-fb03-47a6-a641-aa21cadd652a response_code = 200 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be uri = /5050fe14-0845-40b2-872d-d70a5b2b56be
26262024-08-23T21:59:26.193ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:57601
26272024-08-23T21:59:26.193ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 59 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:57601 req_id = a9907fc8-b2c2-41d1-a579-ee38a1c2ef9b response_code = 200 uri = /baseboard
26282024-08-23T21:59:30.797ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:55384
26292024-08-23T21:59:30.798ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 984 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:55384 req_id = 7279c95d-155e-4e5a-9c2b-8ce7ec9d7891 response_code = 200 uri = /vpc-routes
26302024-08-23T21:59:30.805ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 115 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:55384 req_id = d32481ca-b2c9-4c87-b867-5578a7eb22d5 response_code = 204 uri = /vpc-routes
26312024-08-23T21:59:30.818ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:46483
26322024-08-23T21:59:30.819ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 101 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:46483 req_id = b92467d1-540e-48df-a8f7-12e5691502e7 response_code = 200 uri = /v2p
26332024-08-23T21:59:31.133ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:48711
26342024-08-23T21:59:31.133ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
26352024-08-23T21:59:31.180ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
26362024-08-23T21:59:31.261ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 127514 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:48711 req_id = 7d68cb4f-fb71-45ca-bc0b-41ab13f339c6 response_code = 204 uri = /switch-ports
26372024-08-23T21:59:31.261ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 118 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:48711 req_id = 4dc8f153-b3f1-49e5-94e3-48e9e23875be response_code = 200 uri = /network-bootstore-config
26382024-08-23T21:59:31.370ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:47593
26392024-08-23T21:59:31.370ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 177 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:47593 req_id = 93bc566d-a9fd-480b-aa67-9e1cc44bbd8a response_code = 200 uri = /v2p
26402024-08-23T21:59:31.443ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33544
26412024-08-23T21:59:31.444ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 111 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:33544 req_id = cadf301f-f060-4779-b78f-b504f6d706ae response_code = 200 uri = /v2p
26422024-08-23T21:59:31.488ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:38023
26432024-08-23T21:59:31.488ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 53 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:38023 req_id = af3848ab-4701-4e86-b2ba-096b40802e5b response_code = 200 uri = /vpc-routes
26442024-08-23T21:59:31.496ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 868 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:38023 req_id = bb91e5f7-eba6-4f85-8a8d-71272d2b6788 response_code = 204 uri = /vpc-routes
26452024-08-23T21:59:31.550ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61805
26462024-08-23T21:59:31.550ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 40 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:61805 req_id = 6c1f91b8-7454-4323-9dc9-b6af6b20d636 response_code = 200 uri = /vpc-routes
26472024-08-23T21:59:31.556ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 81 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:61805 req_id = 4552a147-d495-4a1c-bb84-b3a817f359dd response_code = 204 uri = /vpc-routes
26482024-08-23T21:59:31.736ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:51598
26492024-08-23T21:59:31.736ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
26502024-08-23T21:59:31.778ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
26512024-08-23T21:59:31.839ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 102918 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:51598 req_id = fa15e336-b4c1-428d-99fe-be7d5d269f00 response_code = 204 uri = /switch-ports
26522024-08-23T21:59:31.839ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:51598 req_id = a2edcf11-a13e-4af4-89c4-8a6de43cef85 response_code = 200 uri = /network-bootstore-config
26532024-08-23T21:59:31.861ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:58950
26542024-08-23T21:59:31.861ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
26552024-08-23T21:59:31.901ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
26562024-08-23T21:59:31.962ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 101061 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:58950 req_id = fd604dcf-98eb-4da9-8c29-0f864ee27493 response_code = 204 uri = /switch-ports
26572024-08-23T21:59:31.962ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 112 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:58950 req_id = a9876ee7-1094-4817-b279-26e076826304 response_code = 200 uri = /network-bootstore-config
26582024-08-23T21:59:39.226ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6364 local_addr = [fd00:1122:3344:101::1]:46625 method = GET remote_addr = [fd00:1122:3344:101::d]:62985 req_id = 37132e87-105e-4036-854e-7d02bc8ab99b response_code = 200 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be uri = /5050fe14-0845-40b2-872d-d70a5b2b56be
26592024-08-23T21:59:56.230ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:63500
26602024-08-23T21:59:56.230ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 553 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:63500 req_id = 55944721-28dc-4a37-a17f-797a99d067cf response_code = 200 uri = /baseboard
26612024-08-23T22:00:00.747ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:34896
26622024-08-23T22:00:00.747ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 66 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:34896 req_id = 368ac300-74ac-4b97-8bc4-5c7fba81cdfa response_code = 200 uri = /vpc-routes
26632024-08-23T22:00:00.756ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:34896 req_id = d75bc3b3-bf55-498c-b9f1-fae47b38872c response_code = 204 uri = /vpc-routes
26642024-08-23T22:00:00.910ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:48257
26652024-08-23T22:00:00.911ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1386 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:48257 req_id = ab1ae818-6e21-43e9-839b-b2d03678a9c1 response_code = 200 uri = /v2p
26662024-08-23T22:00:01.090ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:56913
26672024-08-23T22:00:01.090ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
26682024-08-23T22:00:01.150ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
26692024-08-23T22:00:01.299ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 208478 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:56913 req_id = 16e99220-e1c3-4821-b152-5f77afd3e1ad response_code = 204 uri = /switch-ports
26702024-08-23T22:00:01.299ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:43168
26712024-08-23T22:00:01.299ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 115 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:43168 req_id = 89be85bd-c431-49ac-9447-30b5aa8a2505 response_code = 200 uri = /vpc-routes
26722024-08-23T22:00:01.299ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 232 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:56913 req_id = 72109ad0-0a83-4523-9387-86de31832a50 response_code = 200 uri = /network-bootstore-config
26732024-08-23T22:00:01.308ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 115 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:43168 req_id = b1f2de5f-8d8f-439e-a2bf-91b2b9887f98 response_code = 204 uri = /vpc-routes
26742024-08-23T22:00:01.310ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63671
26752024-08-23T22:00:01.310ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 32 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:63671 req_id = 85679d77-52de-4030-ac68-b85ef6892063 response_code = 200 uri = /vpc-routes
26762024-08-23T22:00:01.319ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 76 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:63671 req_id = 2d7168a8-893e-4a43-a5ae-5ebd8ea0c7fe response_code = 204 uri = /vpc-routes
26772024-08-23T22:00:01.319ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:47417
26782024-08-23T22:00:01.320ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1121 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:47417 req_id = b09bd784-8437-4497-82af-04e3a299d98d response_code = 200 uri = /v2p
26792024-08-23T22:00:01.425ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41236
26802024-08-23T22:00:01.425ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 134 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41236 req_id = e4abdb91-7875-4d89-9879-e2f7bc94dedb response_code = 200 uri = /v2p
26812024-08-23T22:00:01.490ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:50843
26822024-08-23T22:00:01.490ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
26832024-08-23T22:00:01.578ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
26842024-08-23T22:00:01.718ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 227882 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:50843 req_id = f7546ce0-37ba-4721-98ff-d2e9dc33d465 response_code = 204 uri = /switch-ports
26852024-08-23T22:00:01.718ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:50326
26862024-08-23T22:00:01.718ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
26872024-08-23T22:00:01.718ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 257 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:50843 req_id = 85587396-d9d1-4754-a610-e152ad2276eb response_code = 200 uri = /network-bootstore-config
26882024-08-23T22:00:01.801ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
26892024-08-23T22:00:01.929ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 211150 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:50326 req_id = 1750f958-8097-4fe3-a2f1-7330ce279ad0 response_code = 204 uri = /switch-ports
26902024-08-23T22:00:01.930ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 168 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:50326 req_id = a5df4eca-0677-4602-899d-a18a39c83560 response_code = 200 uri = /network-bootstore-config
26912024-08-23T22:00:09.238ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7917 local_addr = [fd00:1122:3344:101::1]:46625 method = GET remote_addr = [fd00:1122:3344:101::d]:62985 req_id = 54aaccd8-3290-40c4-9987-82e22e38b3ad response_code = 200 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be uri = /5050fe14-0845-40b2-872d-d70a5b2b56be
26922024-08-23T22:00:26.263ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:63073
26932024-08-23T22:00:26.263ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 57 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:63073 req_id = 1386fa99-0632-4d2c-9f21-f47b8480fa3a response_code = 200 uri = /baseboard
26942024-08-23T22:00:30.797ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:42824
26952024-08-23T22:00:30.797ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 178 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:42824 req_id = 622bd161-1d1c-4896-96a6-572956825dad response_code = 200 uri = /v2p
26962024-08-23T22:00:30.889ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:41689
26972024-08-23T22:00:30.889ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 57 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:41689 req_id = ad2df47d-9b07-4a3e-90d3-39f6d4dece58 response_code = 200 uri = /vpc-routes
26982024-08-23T22:00:30.896ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 103 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:41689 req_id = 12436475-5014-4278-8b82-a6821f5b48f7 response_code = 204 uri = /vpc-routes
26992024-08-23T22:00:31.040ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:34909
27002024-08-23T22:00:31.041ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
27012024-08-23T22:00:31.084ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
27022024-08-23T22:00:31.154ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 113907 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:34909 req_id = 727bf345-ddde-4257-b18f-74f7620998cd response_code = 204 uri = /switch-ports
27032024-08-23T22:00:31.155ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 120 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:34909 req_id = 7884615f-32f9-4f96-952b-de56abff73c4 response_code = 200 uri = /network-bootstore-config
27042024-08-23T22:00:31.354ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:64482
27052024-08-23T22:00:31.354ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 113 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:64482 req_id = 86d88942-19b0-464d-8ecc-b89017766c55 response_code = 200 uri = /vpc-routes
27062024-08-23T22:00:31.361ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 123 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:64482 req_id = 983d6885-6ec2-46d6-8620-083dfafa2e4a response_code = 204 uri = /vpc-routes
27072024-08-23T22:00:31.542ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:57958
27082024-08-23T22:00:31.542ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 123 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:57958 req_id = faf75f59-1ef5-4a12-95ee-f9e76f1c89e8 response_code = 200 uri = /v2p
27092024-08-23T22:00:31.571ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:55956
27102024-08-23T22:00:31.571ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 94 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:55956 req_id = 5d713121-a546-4c6a-a711-6e4e6ebe949e response_code = 200 uri = /v2p
27112024-08-23T22:00:31.668ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:50283
27122024-08-23T22:00:31.668ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 82 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:50283 req_id = 89be7e95-d6d0-48f3-bd34-e20c54aa8e0e response_code = 200 uri = /vpc-routes
27132024-08-23T22:00:31.675ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 917 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:50283 req_id = c3a25339-2c32-4c33-bbc9-ff1d375d2eb0 response_code = 204 uri = /vpc-routes
27142024-08-23T22:00:31.695ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:46407
27152024-08-23T22:00:31.695ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
27162024-08-23T22:00:31.735ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
27172024-08-23T22:00:31.790ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 95014 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:46407 req_id = 3df64379-d2c7-4e12-9a33-f8d114e7923b response_code = 204 uri = /switch-ports
27182024-08-23T22:00:31.790ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:50030
27192024-08-23T22:00:31.791ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
27202024-08-23T22:00:31.794ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 3583 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:46407 req_id = 93bba67d-0d2d-4df5-9f4f-4e4b37bd2101 response_code = 200 uri = /network-bootstore-config
27212024-08-23T22:00:31.830ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
27222024-08-23T22:00:31.887ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 96626 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:50030 req_id = d5d09ec8-b2a6-4843-9167-4ca7af45f689 response_code = 204 uri = /switch-ports
27232024-08-23T22:00:31.888ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 180 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:50030 req_id = 78024fa8-de22-4b9a-8ecc-e81975964173 response_code = 200 uri = /network-bootstore-config
27242024-08-23T22:00:39.279ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6846 local_addr = [fd00:1122:3344:101::1]:46625 method = GET remote_addr = [fd00:1122:3344:101::d]:62985 req_id = 11780133-9675-4471-8ab0-a8b1688702c9 response_code = 200 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be uri = /5050fe14-0845-40b2-872d-d70a5b2b56be
27252024-08-23T22:00:56.297ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:59689
27262024-08-23T22:00:56.297ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 44 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:59689 req_id = a1c3f2cd-9449-436f-8e16-5c1be5e8bb3e response_code = 200 uri = /baseboard
27272024-08-23T22:01:00.700ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:57241
27282024-08-23T22:01:00.700ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 86 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:57241 req_id = ba974787-ce52-431e-a137-800ea90f8d14 response_code = 200 uri = /vpc-routes
27292024-08-23T22:01:00.707ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 126 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:57241 req_id = 613bd0ef-b19c-41da-b7b1-2ce3f719e9b4 response_code = 204 uri = /vpc-routes
27302024-08-23T22:01:00.820ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:45309
27312024-08-23T22:01:00.820ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 812 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:45309 req_id = 18d666ec-8f0b-4b25-8499-4b6aa7ec27ca response_code = 200 uri = /v2p
27322024-08-23T22:01:00.959ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:36439
27332024-08-23T22:01:00.959ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
27342024-08-23T22:01:00.999ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
27352024-08-23T22:01:01.056ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 97100 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:36439 req_id = 19e5af2d-ac66-496a-acda-d167e1f4b483 response_code = 204 uri = /switch-ports
27362024-08-23T22:01:01.057ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 101 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:36439 req_id = b6458829-e2f0-4b27-8ee8-3ca23f09289f response_code = 200 uri = /network-bootstore-config
27372024-08-23T22:01:01.236ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40177
27382024-08-23T22:01:01.236ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 118 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40177 req_id = 1347d64d-8341-4a5b-9f0c-7524918ff0f3 response_code = 200 uri = /vpc-routes
27392024-08-23T22:01:01.244ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 129 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:40177 req_id = 6030c7d3-11f1-442a-b063-b4e46f1cbd15 response_code = 204 uri = /vpc-routes
27402024-08-23T22:01:01.347ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:53153
27412024-08-23T22:01:01.348ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1053 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:53153 req_id = 6bae71c4-9028-4cda-ab86-3d3d124e71fb response_code = 200 uri = /v2p
27422024-08-23T22:01:01.384ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:46808
27432024-08-23T22:01:01.384ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 87 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:46808 req_id = 4f436dc5-e0a4-48f8-aa42-630f4afb8b28 response_code = 200 uri = /v2p
27442024-08-23T22:01:01.401ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41508
27452024-08-23T22:01:01.402ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 57 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41508 req_id = 6bec0e7b-446f-4231-afe7-7cc4561a752b response_code = 200 uri = /vpc-routes
27462024-08-23T22:01:01.408ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:41508 req_id = e9e78032-370a-429e-bdac-950f62235d1c response_code = 204 uri = /vpc-routes
27472024-08-23T22:01:01.439ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:54582
27482024-08-23T22:01:01.439ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
27492024-08-23T22:01:01.475ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
27502024-08-23T22:01:01.529ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 89644 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:54582 req_id = db94c4b7-c175-40e9-a593-6591c3b47615 response_code = 204 uri = /switch-ports
27512024-08-23T22:01:01.529ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:40222
27522024-08-23T22:01:01.529ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
27532024-08-23T22:01:01.564ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
27542024-08-23T22:01:01.621ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 91728 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:40222 req_id = e52c832d-20f0-49cf-b526-aa5296712a05 response_code = 204 uri = /switch-ports
27552024-08-23T22:01:01.621ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 141 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:54582 req_id = 57c808d9-397b-4bf3-8cab-7bfc606f2f19 response_code = 200 uri = /network-bootstore-config
27562024-08-23T22:01:01.621ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 203 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:40222 req_id = b7ef718b-47ed-4bbf-bea8-99da2485d1c4 response_code = 200 uri = /network-bootstore-config
27572024-08-23T22:01:09.279ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 5898 local_addr = [fd00:1122:3344:101::1]:46625 method = GET remote_addr = [fd00:1122:3344:101::d]:62985 req_id = 2d8299f8-dba6-48c7-98a6-936d22663f5b response_code = 200 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be uri = /5050fe14-0845-40b2-872d-d70a5b2b56be
27582024-08-23T22:01:26.331ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:64423
27592024-08-23T22:01:26.331ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 675 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:64423 req_id = 7441a8e9-e90d-4fb0-b7b3-c72dd928c3f7 response_code = 200 uri = /baseboard
27602024-08-23T22:01:30.776ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:44062
27612024-08-23T22:01:30.776ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 63 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:44062 req_id = 48a2f99a-f27b-437b-a1a2-e610c940fa6b response_code = 200 uri = /vpc-routes
27622024-08-23T22:01:30.784ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 751 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:44062 req_id = 9fb6be63-c910-4b58-8f08-a98915deb538 response_code = 204 uri = /vpc-routes
27632024-08-23T22:01:30.807ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:33540
27642024-08-23T22:01:30.808ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 889 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:33540 req_id = deb1d0bc-d61b-471c-a343-a2f34434227f response_code = 200 uri = /v2p
27652024-08-23T22:01:31.242ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:47972
27662024-08-23T22:01:31.242ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
27672024-08-23T22:01:31.283ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
27682024-08-23T22:01:31.342ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 100067 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:47972 req_id = 7b071990-65e0-4a10-85f0-67cb8aeb96ea response_code = 204 uri = /switch-ports
27692024-08-23T22:01:31.342ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:63657
27702024-08-23T22:01:31.343ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:50034
27712024-08-23T22:01:31.343ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 203 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:63657 req_id = 00dbe5c9-a2a3-4f5d-abb5-f215f53e69be response_code = 200 uri = /v2p
27722024-08-23T22:01:31.344ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1247 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:50034 req_id = 017ce25b-221f-4270-aed0-21c35da2d941 response_code = 200 uri = /vpc-routes
27732024-08-23T22:01:31.344ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1373 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:47972 req_id = 4d8c7060-f3ac-4999-929e-6618e62c0696 response_code = 200 uri = /network-bootstore-config
27742024-08-23T22:01:31.352ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 141 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:50034 req_id = 92fd3f3b-97f1-4b27-9d2d-01bcddb5978a response_code = 204 uri = /vpc-routes
27752024-08-23T22:01:31.437ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:48984
27762024-08-23T22:01:31.437ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 98 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:48984 req_id = 0ebdf725-b7be-4064-a490-72424f40deda response_code = 200 uri = /v2p
27772024-08-23T22:01:31.554ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47020
27782024-08-23T22:01:31.554ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 52 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47020 req_id = dcf43c44-0bbd-43da-89f1-b23f17566f0f response_code = 200 uri = /vpc-routes
27792024-08-23T22:01:31.560ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 89 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:47020 req_id = 377898bd-d706-4b91-a38c-1fa3e83dbc67 response_code = 204 uri = /vpc-routes
27802024-08-23T22:01:31.581ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36087
27812024-08-23T22:01:31.581ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
27822024-08-23T22:01:31.622ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
27832024-08-23T22:01:31.683ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 102156 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:36087 req_id = a7fdd03e-d288-48e1-8111-58df2d3c3f71 response_code = 204 uri = /switch-ports
27842024-08-23T22:01:31.684ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 976 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:36087 req_id = 08422356-81a5-4bbe-93a8-85861236105e response_code = 200 uri = /network-bootstore-config
27852024-08-23T22:01:31.828ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:38090
27862024-08-23T22:01:31.829ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
27872024-08-23T22:01:31.869ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
27882024-08-23T22:01:31.931ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 102249 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:38090 req_id = cdbb9b2b-0cf2-4399-8ccb-f75bcf68f1d7 response_code = 204 uri = /switch-ports
27892024-08-23T22:01:31.932ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 731 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:38090 req_id = 1cbd7d67-e0a2-41a6-9574-9ff25abfa85e response_code = 200 uri = /network-bootstore-config
27902024-08-23T22:01:39.279ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6182 local_addr = [fd00:1122:3344:101::1]:46625 method = GET remote_addr = [fd00:1122:3344:101::d]:62985 req_id = 06be3171-1d18-4e88-877c-5e970f3266c3 response_code = 200 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be uri = /5050fe14-0845-40b2-872d-d70a5b2b56be
27912024-08-23T22:01:56.364ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:48486
27922024-08-23T22:01:56.364ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 45 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:48486 req_id = 0a691247-56e9-4954-8253-70f46e9a9dfb response_code = 200 uri = /baseboard
27932024-08-23T22:01:59.415ZINFOSledAgent (auto-cleanup-task): running automatic periodic zone bundle cleanup file = sled-agent/src/zone_bundle.rs:197
27942024-08-23T22:01:59.416ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
27952024-08-23T22:01:59.459ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:186 next_cleanup = Instant { tv_sec: 2670, tv_nsec: 206395128 } time_to_next_cleanup = 599.99999408s
27962024-08-23T22:02:00.708ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:40629
27972024-08-23T22:02:00.708ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 101 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:40629 req_id = 559429c1-03af-42b1-a7e2-6cdfea14817a response_code = 200 uri = /vpc-routes
27982024-08-23T22:02:00.709ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:48356
27992024-08-23T22:02:00.710ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 795 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:48356 req_id = 0bc4d1cc-2cd6-4497-8e0e-75bfb07ab518 response_code = 200 uri = /v2p
28002024-08-23T22:02:00.715ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 100 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:40629 req_id = 1ea2f9c7-8030-478a-b076-299163b8f379 response_code = 204 uri = /vpc-routes
28012024-08-23T22:02:00.873ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:36220
28022024-08-23T22:02:00.874ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
28032024-08-23T22:02:00.913ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
28042024-08-23T22:02:00.968ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 94371 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:36220 req_id = c2bb71c7-e3f7-428a-b217-eebc05ffab20 response_code = 204 uri = /switch-ports
28052024-08-23T22:02:00.969ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 733 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:36220 req_id = df777377-ece6-47c4-a36d-d34889403b77 response_code = 200 uri = /network-bootstore-config
28062024-08-23T22:02:01.235ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36621
28072024-08-23T22:02:01.235ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:36621 req_id = 68706430-f158-4120-a08f-304569d648da response_code = 200 uri = /vpc-routes
28082024-08-23T22:02:01.243ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 874 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:36621 req_id = d04b932b-d8d7-4946-baf4-8571666e60b3 response_code = 204 uri = /vpc-routes
28092024-08-23T22:02:01.285ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:33536
28102024-08-23T22:02:01.285ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 137 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:33536 req_id = 0aaff43e-a985-4dcd-9bf1-bce81447e5b9 response_code = 200 uri = /v2p
28112024-08-23T22:02:01.377ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33321
28122024-08-23T22:02:01.377ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 131 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:33321 req_id = 542f00eb-d5e0-478f-8bbc-a03b31219b3d response_code = 200 uri = /v2p
28132024-08-23T22:02:01.434ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61854
28142024-08-23T22:02:01.435ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 57 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:61854 req_id = f369963f-2a33-473d-b902-40f4b4c8952d response_code = 200 uri = /vpc-routes
28152024-08-23T22:02:01.440ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 85 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:61854 req_id = 29bef824-994c-4dd3-9fb0-0bde4b952025 response_code = 204 uri = /vpc-routes
28162024-08-23T22:02:01.445ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58106
28172024-08-23T22:02:01.445ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
28182024-08-23T22:02:01.485ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
28192024-08-23T22:02:01.543ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 98484 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:58106 req_id = ed0a945e-c911-4dc1-b842-bffc53c4ac8e response_code = 204 uri = /switch-ports
28202024-08-23T22:02:01.543ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57449
28212024-08-23T22:02:01.544ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
28222024-08-23T22:02:01.548ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 4320 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58106 req_id = 5aab92fc-b8d1-42f6-a235-b3aa457460c1 response_code = 200 uri = /network-bootstore-config
28232024-08-23T22:02:01.583ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
28242024-08-23T22:02:01.643ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 99689 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:57449 req_id = e056727f-dbfb-4add-8762-0e226c7fae56 response_code = 204 uri = /switch-ports
28252024-08-23T22:02:01.644ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 804 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:57449 req_id = 580c2f7b-9f6a-449a-a590-ccb1634920da response_code = 200 uri = /network-bootstore-config
28262024-08-23T22:02:09.280ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6585 local_addr = [fd00:1122:3344:101::1]:46625 method = GET remote_addr = [fd00:1122:3344:101::d]:62985 req_id = 915bc01d-b23e-4117-8ab8-6470411fd0b2 response_code = 200 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be uri = /5050fe14-0845-40b2-872d-d70a5b2b56be
28272024-08-23T22:02:13.509ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
28282024-08-23T22:02:26.399ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:54055
28292024-08-23T22:02:26.399ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 45 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:54055 req_id = 1c7be9e7-92e3-4132-8f1d-04eb12d016f8 response_code = 200 uri = /baseboard
28302024-08-23T22:02:30.758ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:56888
28312024-08-23T22:02:30.758ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 149 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:56888 req_id = bb4d30a0-204f-411e-8c2e-4cdfb305ffe8 response_code = 200 uri = /v2p
28322024-08-23T22:02:30.798ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:48134
28332024-08-23T22:02:30.798ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 74 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:48134 req_id = 37335ceb-c633-496f-a716-5fdab7e61070 response_code = 200 uri = /vpc-routes
28342024-08-23T22:02:30.805ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 106 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:48134 req_id = 40e0d441-03be-4db5-a5b9-b6bd5dd011f1 response_code = 204 uri = /vpc-routes
28352024-08-23T22:02:30.919ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:61256
28362024-08-23T22:02:30.920ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 2a9b2cf7-2fb3-4597-871e-44332b4d82f2, kind: Service { id: 1db57e91-9837-462c-9f2c-1b4fca780b7e }, name: Name("external-dns-1db57e91-9837-462c-9f2c-1b4fca780b7e"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 231, 24])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 5d1239c9-696e-4616-a2a6-82968e91a2e1, kind: Service { id: b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac }, name: Name("external-dns-b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 184])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 00000426-1f68-402a-a60e-4fa2f73c115f, kind: Service { id: 4e26237f-8cdc-4da1-b44a-9b00d20bd873 }, name: Name("nexus-4e26237f-8cdc-4da1-b44a-9b00d20bd873"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 190, 168])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 5cf061ea-987b-4986-8378-702c868c90c5, kind: Service { id: f9a2f81c-00cf-49a5-a96e-1c4dfeba1728 }, name: Name("nexus-f9a2f81c-00cf-49a5-a96e-1c4dfeba1728"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 196, 138])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: b2284d48-b022-4af4-a2f6-3a691122eb62, kind: Service { id: 2c94012a-643d-4f61-8185-c8786e00e320 }, name: Name("nexus-2c94012a-643d-4f61-8185-c8786e00e320"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 253, 42])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
28372024-08-23T22:02:30.920ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
28382024-08-23T22:02:30.920ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
28392024-08-23T22:02:30.920ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
28402024-08-23T22:02:30.920ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
28412024-08-23T22:02:30.920ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
28422024-08-23T22:02:30.920ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
28432024-08-23T22:02:30.920ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 349 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:61256 req_id = 2ca00716-b9e1-4428-b708-374e2ca29b84 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
28442024-08-23T22:02:31.196ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:48620
28452024-08-23T22:02:31.197ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
28462024-08-23T22:02:31.249ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
28472024-08-23T22:02:31.322ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 126450 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:48620 req_id = 1b328dae-a8cd-4cd7-a373-b1067bb031d5 response_code = 204 uri = /switch-ports
28482024-08-23T22:02:31.323ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 226 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:48620 req_id = 32f272da-c7f6-4ab6-97f1-f4f3e650b4dd response_code = 200 uri = /network-bootstore-config
28492024-08-23T22:02:31.378ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40138
28502024-08-23T22:02:31.378ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 83 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40138 req_id = c8eff402-ec69-4cd8-9688-f3553bdb6991 response_code = 200 uri = /vpc-routes
28512024-08-23T22:02:31.385ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 112 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:40138 req_id = 0a99da12-0735-4cea-8630-48aa92686930 response_code = 204 uri = /vpc-routes
28522024-08-23T22:02:31.390ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:55769
28532024-08-23T22:02:31.390ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 116 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:55769 req_id = 5f4db4fb-fa8f-4db5-9eff-0d5d7cd10184 response_code = 200 uri = /v2p
28542024-08-23T22:02:31.410ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:53771
28552024-08-23T22:02:31.410ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 2a9b2cf7-2fb3-4597-871e-44332b4d82f2, kind: Service { id: 1db57e91-9837-462c-9f2c-1b4fca780b7e }, name: Name("external-dns-1db57e91-9837-462c-9f2c-1b4fca780b7e"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 231, 24])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 5d1239c9-696e-4616-a2a6-82968e91a2e1, kind: Service { id: b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac }, name: Name("external-dns-b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 184])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 00000426-1f68-402a-a60e-4fa2f73c115f, kind: Service { id: 4e26237f-8cdc-4da1-b44a-9b00d20bd873 }, name: Name("nexus-4e26237f-8cdc-4da1-b44a-9b00d20bd873"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 190, 168])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 5cf061ea-987b-4986-8378-702c868c90c5, kind: Service { id: f9a2f81c-00cf-49a5-a96e-1c4dfeba1728 }, name: Name("nexus-f9a2f81c-00cf-49a5-a96e-1c4dfeba1728"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 196, 138])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: b2284d48-b022-4af4-a2f6-3a691122eb62, kind: Service { id: 2c94012a-643d-4f61-8185-c8786e00e320 }, name: Name("nexus-2c94012a-643d-4f61-8185-c8786e00e320"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 253, 42])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
28562024-08-23T22:02:31.410ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
28572024-08-23T22:02:31.410ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
28582024-08-23T22:02:31.410ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
28592024-08-23T22:02:31.410ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
28602024-08-23T22:02:31.410ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
28612024-08-23T22:02:31.410ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
28622024-08-23T22:02:31.411ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 275 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:53771 req_id = 38f3a84a-c47e-457d-8d26-3fca731f86c3 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
28632024-08-23T22:02:31.450ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:46372
28642024-08-23T22:02:31.450ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:46372 req_id = 89d310c5-b916-4a76-87ef-0f6989392be1 response_code = 200 uri = /v2p
28652024-08-23T22:02:31.460ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42688
28662024-08-23T22:02:31.460ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 37 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:42688 req_id = 346e21c5-1578-48ac-a8db-c1f044c7e95d response_code = 200 uri = /vpc-routes
28672024-08-23T22:02:31.467ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 74 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:42688 req_id = 0236898b-078b-4126-9acf-5c43732f5c60 response_code = 204 uri = /vpc-routes
28682024-08-23T22:02:31.671ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61852
28692024-08-23T22:02:31.671ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 2a9b2cf7-2fb3-4597-871e-44332b4d82f2, kind: Service { id: 1db57e91-9837-462c-9f2c-1b4fca780b7e }, name: Name("external-dns-1db57e91-9837-462c-9f2c-1b4fca780b7e"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 231, 24])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 5d1239c9-696e-4616-a2a6-82968e91a2e1, kind: Service { id: b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac }, name: Name("external-dns-b28fec8f-a1e5-40b2-a907-cc5bb6eb8cac"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 184])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 00000426-1f68-402a-a60e-4fa2f73c115f, kind: Service { id: 4e26237f-8cdc-4da1-b44a-9b00d20bd873 }, name: Name("nexus-4e26237f-8cdc-4da1-b44a-9b00d20bd873"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 190, 168])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 5cf061ea-987b-4986-8378-702c868c90c5, kind: Service { id: f9a2f81c-00cf-49a5-a96e-1c4dfeba1728 }, name: Name("nexus-f9a2f81c-00cf-49a5-a96e-1c4dfeba1728"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 196, 138])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: b2284d48-b022-4af4-a2f6-3a691122eb62, kind: Service { id: 2c94012a-643d-4f61-8185-c8786e00e320 }, name: Name("nexus-2c94012a-643d-4f61-8185-c8786e00e320"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 253, 42])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
28702024-08-23T22:02:31.671ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
28712024-08-23T22:02:31.671ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
28722024-08-23T22:02:31.671ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
28732024-08-23T22:02:31.671ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
28742024-08-23T22:02:31.671ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
28752024-08-23T22:02:31.671ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
28762024-08-23T22:02:31.671ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 307 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:61852 req_id = 1724bc1e-7814-4e02-9926-d07818149946 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
28772024-08-23T22:02:31.675ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:49458
28782024-08-23T22:02:31.675ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
28792024-08-23T22:02:31.716ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
28802024-08-23T22:02:31.776ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 100458 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:49458 req_id = 9a14ba3a-5b88-4f53-a1cd-0167a2865013 response_code = 204 uri = /switch-ports
28812024-08-23T22:02:31.777ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 735 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:49458 req_id = feb6767c-91f5-4078-8e94-888e5ac491d2 response_code = 200 uri = /network-bootstore-config
28822024-08-23T22:02:31.916ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:35609
28832024-08-23T22:02:31.917ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
28842024-08-23T22:02:31.956ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
28852024-08-23T22:02:32.012ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 95911 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:35609 req_id = dffd2df6-8976-4bbe-a1ae-2e52579173f1 response_code = 204 uri = /switch-ports
28862024-08-23T22:02:32.013ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 105 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:35609 req_id = 5e7d744a-6f6a-4ff1-b73d-2f9485c80adf response_code = 200 uri = /network-bootstore-config
28872024-08-23T22:02:39.280ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6434 local_addr = [fd00:1122:3344:101::1]:46625 method = GET remote_addr = [fd00:1122:3344:101::d]:62985 req_id = df0d9f71-51ef-44a2-af80-b3ed15f051b1 response_code = 200 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be uri = /5050fe14-0845-40b2-872d-d70a5b2b56be
28882024-08-23T22:02:56.434ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:65403
28892024-08-23T22:02:56.434ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 42 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:65403 req_id = 8c2aa861-db27-4fcd-baf6-00c72c3a2c89 response_code = 200 uri = /baseboard
28902024-08-23T22:03:00.726ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:48531
28912024-08-23T22:03:00.726ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 143 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:48531 req_id = 01796382-3600-4d7f-999a-4ffc520beffe response_code = 200 uri = /v2p
28922024-08-23T22:03:00.877ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50317
28932024-08-23T22:03:00.877ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 40 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50317 req_id = 759fff6e-f77d-4503-a603-3741a17ca760 response_code = 200 uri = /vpc-routes
28942024-08-23T22:03:00.883ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 76 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:50317 req_id = 425438ce-65f8-48e1-9985-5806bc284060 response_code = 204 uri = /vpc-routes
28952024-08-23T22:03:00.955ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:43888
28962024-08-23T22:03:00.955ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
28972024-08-23T22:03:00.993ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
28982024-08-23T22:03:01.046ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 91386 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:43888 req_id = 7713d984-b139-4d83-820d-a1cba1700017 response_code = 204 uri = /switch-ports
28992024-08-23T22:03:01.046ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:43888 req_id = 477dc818-d12f-4c5b-ade4-a12a79216493 response_code = 200 uri = /network-bootstore-config
29002024-08-23T22:03:01.294ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37514
29012024-08-23T22:03:01.294ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 129 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:37514 req_id = cb54e940-ea4f-438e-943f-d348b2e7605c response_code = 200 uri = /vpc-routes
29022024-08-23T22:03:01.299ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:44120
29032024-08-23T22:03:01.300ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 837 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:44120 req_id = e32f908d-529f-4c7f-ba9c-b25d9d288732 response_code = 200 uri = /vpc-routes
29042024-08-23T22:03:01.301ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 120 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:37514 req_id = 9e04c136-de3f-4831-a936-61a88a09fbf6 response_code = 204 uri = /vpc-routes
29052024-08-23T22:03:01.306ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 65 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:44120 req_id = b5dafc8e-2256-4405-896e-e91c63cef18f response_code = 204 uri = /vpc-routes
29062024-08-23T22:03:01.315ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:51409
29072024-08-23T22:03:01.315ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 126 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:51409 req_id = 2cf2cf66-df8a-4676-a37a-e4b238adbe26 response_code = 200 uri = /v2p
29082024-08-23T22:03:01.321ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57617
29092024-08-23T22:03:01.321ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 74 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:57617 req_id = 029d2fd8-97c7-4915-a61e-b2682c1783c6 response_code = 200 uri = /v2p
29102024-08-23T22:03:01.454ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:49535
29112024-08-23T22:03:01.454ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
29122024-08-23T22:03:01.493ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
29132024-08-23T22:03:01.555ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 101476 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:49535 req_id = 155eb95c-49b3-4c29-a32e-9827cbecf527 response_code = 204 uri = /switch-ports
29142024-08-23T22:03:01.555ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:34975
29152024-08-23T22:03:01.557ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
29162024-08-23T22:03:01.557ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1219 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:49535 req_id = 3bb07a53-60d0-4515-8ce0-5d955a607708 response_code = 200 uri = /network-bootstore-config
29172024-08-23T22:03:01.593ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
29182024-08-23T22:03:01.649ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 93397 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:34975 req_id = 602b3970-34ae-48e6-8ece-d53ed9fab389 response_code = 204 uri = /switch-ports
29192024-08-23T22:03:01.650ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 659 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:34975 req_id = 2771685a-dc25-4dd8-803b-a79533cda86e response_code = 200 uri = /network-bootstore-config
29202024-08-23T22:03:09.394ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7130 local_addr = [fd00:1122:3344:101::1]:46625 method = GET remote_addr = [fd00:1122:3344:101::d]:62985 req_id = 8d6350df-403d-4b3d-bc7c-7739873e52f0 response_code = 200 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be uri = /5050fe14-0845-40b2-872d-d70a5b2b56be
29212024-08-23T22:03:26.472ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:63333
29222024-08-23T22:03:26.472ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 56 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:63333 req_id = 6f7842e8-1b68-484e-9b1d-01a591ca4157 response_code = 200 uri = /baseboard
29232024-08-23T22:03:30.794ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:46104
29242024-08-23T22:03:30.794ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 94 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:46104 req_id = 76210876-20c7-4f37-80db-b7927d850990 response_code = 200 uri = /vpc-routes
29252024-08-23T22:03:30.800ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 120 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:46104 req_id = bf1569c8-4503-47c0-b8cc-ba613495db4f response_code = 204 uri = /vpc-routes
29262024-08-23T22:03:30.936ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37983
29272024-08-23T22:03:30.936ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 124 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:37983 req_id = 1418f9a4-3372-465f-956e-cea82ed35952 response_code = 200 uri = /v2p
29282024-08-23T22:03:31.283ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:53387
29292024-08-23T22:03:31.283ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
29302024-08-23T22:03:31.327ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
29312024-08-23T22:03:31.390ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 107827 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:53387 req_id = ab136e5a-4857-4e43-b89b-213ba222b8e6 response_code = 204 uri = /switch-ports
29322024-08-23T22:03:31.391ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:52687
29332024-08-23T22:03:31.391ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 203 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:52687 req_id = f57df826-dedb-4309-a510-3a25beb70538 response_code = 200 uri = /v2p
29342024-08-23T22:03:31.391ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 243 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:53387 req_id = 6f300c1b-2eba-4c51-b8d1-b37a9554c3db response_code = 200 uri = /network-bootstore-config
29352024-08-23T22:03:31.404ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36954
29362024-08-23T22:03:31.405ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 943 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:36954 req_id = b4f09ff9-eb67-47b6-a266-f0429259997a response_code = 200 uri = /vpc-routes
29372024-08-23T22:03:31.411ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 129 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:36954 req_id = 18bf7f5a-0af6-4d28-a0a3-350a49b80457 response_code = 204 uri = /vpc-routes
29382024-08-23T22:03:31.419ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:40441
29392024-08-23T22:03:31.419ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 125 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:40441 req_id = 21e1db86-1634-447f-ab4a-bb8617a50249 response_code = 200 uri = /v2p
29402024-08-23T22:03:31.459ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:46704
29412024-08-23T22:03:31.460ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1007 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:46704 req_id = 8c252da0-2ab1-4856-8094-21f806e31804 response_code = 200 uri = /vpc-routes
29422024-08-23T22:03:31.466ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 82 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:46704 req_id = dbbf2619-3537-4294-8072-1f37e3655a7c response_code = 204 uri = /vpc-routes
29432024-08-23T22:03:31.599ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:38421
29442024-08-23T22:03:31.599ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
29452024-08-23T22:03:31.638ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
29462024-08-23T22:03:31.700ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 101019 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:38421 req_id = c0caa9a7-28e4-4b80-b136-5f077f367393 response_code = 204 uri = /switch-ports
29472024-08-23T22:03:31.700ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 95 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:38421 req_id = d31e8671-2b9b-4f17-b427-8a60abd4e018 response_code = 200 uri = /network-bootstore-config
29482024-08-23T22:03:31.719ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47883
29492024-08-23T22:03:31.720ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
29502024-08-23T22:03:31.755ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
29512024-08-23T22:03:31.813ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 94030 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:47883 req_id = e888c2c7-0ce6-4101-9082-6a2d594b23c3 response_code = 204 uri = /switch-ports
29522024-08-23T22:03:31.814ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 231 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47883 req_id = e6a97142-6c45-4faf-9b51-1e313198c268 response_code = 200 uri = /network-bootstore-config
29532024-08-23T22:03:39.393ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 5793 local_addr = [fd00:1122:3344:101::1]:46625 method = GET remote_addr = [fd00:1122:3344:101::d]:62985 req_id = e746fd8d-9c1a-44c8-8295-c664ffc3ad22 response_code = 200 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be uri = /5050fe14-0845-40b2-872d-d70a5b2b56be
29542024-08-23T22:03:56.505ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:44963
29552024-08-23T22:03:56.505ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 40 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:44963 req_id = 208ad112-e21c-4e15-be3b-c8b7bb18bfa0 response_code = 200 uri = /baseboard
29562024-08-23T22:04:00.727ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:51604
29572024-08-23T22:04:00.727ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:51604 req_id = 36f5baaf-1d43-47e3-a3ad-f02623952cc3 response_code = 200 uri = /vpc-routes
29582024-08-23T22:04:00.730ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:42915
29592024-08-23T22:04:00.730ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 111 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:42915 req_id = 6b764507-8f39-4f7b-aaf6-b6964ae83111 response_code = 200 uri = /v2p
29602024-08-23T22:04:00.734ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 616 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:51604 req_id = 3e51df5e-b4a0-465f-aab6-1c118c41d6fc response_code = 204 uri = /vpc-routes
29612024-08-23T22:04:00.958ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:62403
29622024-08-23T22:04:00.958ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
29632024-08-23T22:04:00.998ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
29642024-08-23T22:04:01.058ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 100364 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:62403 req_id = 990a6a47-8278-46ac-86f7-4b76a75fe1d1 response_code = 204 uri = /switch-ports
29652024-08-23T22:04:01.059ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 110 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:62403 req_id = b474e5a8-6bc6-473c-8cc7-9d393837f0a0 response_code = 200 uri = /network-bootstore-config
29662024-08-23T22:04:01.299ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:48662
29672024-08-23T22:04:01.300ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 199 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:48662 req_id = 958a0e9a-dca8-48f1-8cfd-aa3ff7814fa8 response_code = 200 uri = /v2p
29682024-08-23T22:04:01.319ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58964
29692024-08-23T22:04:01.320ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 794 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58964 req_id = 82b972fa-934f-4b66-a0be-9f36a867e4aa response_code = 200 uri = /vpc-routes
29702024-08-23T22:04:01.328ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1031 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:58964 req_id = 6680e80a-7ced-41f9-bca5-6c27f0fed07a response_code = 204 uri = /vpc-routes
29712024-08-23T22:04:01.341ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63963
29722024-08-23T22:04:01.342ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 89 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:63963 req_id = 729e2ad0-8b61-403d-b3ae-3e51237648dc response_code = 200 uri = /v2p
29732024-08-23T22:04:01.423ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:53195
29742024-08-23T22:04:01.423ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 110 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:53195 req_id = 868c9797-5b0d-4f60-96d1-988cfc7f9ad1 response_code = 200 uri = /vpc-routes
29752024-08-23T22:04:01.429ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:53195 req_id = 7802cdb8-2d67-4475-b194-f6cfca0f11f9 response_code = 204 uri = /vpc-routes
29762024-08-23T22:04:01.515ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:56366
29772024-08-23T22:04:01.515ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
29782024-08-23T22:04:01.553ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
29792024-08-23T22:04:01.610ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 95233 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:56366 req_id = 5705f140-2adb-48f9-bd37-91cfc4cbb1ce response_code = 204 uri = /switch-ports
29802024-08-23T22:04:01.611ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:46712
29812024-08-23T22:04:01.611ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
29822024-08-23T22:04:01.648ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
29832024-08-23T22:04:01.710ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 99798 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:46712 req_id = 224a3fb3-8802-4e42-8e6a-c1f10ce08597 response_code = 204 uri = /switch-ports
29842024-08-23T22:04:01.711ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 152 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:56366 req_id = 2e0f11a3-7e20-475d-972a-1a41c20d16e2 response_code = 200 uri = /network-bootstore-config
29852024-08-23T22:04:01.711ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 229 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:46712 req_id = f5cd6740-eb98-497d-8056-69d1347cdb32 response_code = 200 uri = /network-bootstore-config
29862024-08-23T22:04:04.624ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:40808
29872024-08-23T22:04:04.624ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 184 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:40808 req_id = 204315c4-6c28-46e4-a247-600bb2ad1011 response_code = 200 uri = /v2p
29882024-08-23T22:04:04.628ZINFOSledAgent (PortManager): Mapping virtual NIC to physical host file = illumos-utils/src/opte/port_manager.rs:778 mapping = VirtualNetworkInterfaceHost { virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 240, 227, 128])), physical_host_ip: fd00:1122:3344:101::1, vni: Vni(10829320) }
29892024-08-23T22:04:04.628ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1730 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:40808 req_id = 56e2eb00-4eb8-4674-b477-51f3387139fb response_code = 204 uri = /v2p
29902024-08-23T22:04:04.771ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54383
29912024-08-23T22:04:04.783ZINFOSledAgent (InstanceManager): ensuring instance is registered file = sled-agent/src/instance_manager.rs:569 hardware = InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: ab4b9722-63b3-4972-8fe1-3031f72ea0c3, kind: Instance { id: 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 227, 128])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(10829320), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.1.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.1.176), floating_ips: [], firewall_rules: [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: ab4b9722-63b3-4972-8fe1-3031f72ea0c3, kind: Instance { id: 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 227, 128])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(10829320), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: ab4b9722-63b3-4972-8fe1-3031f72ea0c3, kind: Instance { id: 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 227, 128])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(10829320), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(10829320))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: ab4b9722-63b3-4972-8fe1-3031f72ea0c3, kind: Instance { id: 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 227, 128])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(10829320), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-491aa771766", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 5eaa0251-8bc1-48a1-8803-cb58a8bbfe9a, read_only_parent: Some(Volume { block_size: 512, id: 7524761e-59da-402c-a1b9-719097044aad, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 4712ad4e-9156-4dba-9c51-ba0326c965f5, key: Some("jW1c5RNZuQwtt44kNqIs0WM52ryD4P56RmdYsqD+pbI="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::1a]:19002", "[fd00:1122:3344:101::12]:19002", "[fd00:1122:3344:101::15]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 5eaa0251-8bc1-48a1-8803-cb58a8bbfe9a, key: Some("mMBh8eQ+6IHiSbQUR1SypnKTM3vaGv8fE2/Z6KzO1sU="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::13]:19001", "[fd00:1122:3344:101::14]:19000", "[fd00:1122:3344:101::16]:19000"] } }] } }], cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAABJgrAXWRdZAACCsBdZAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAASYKwF1kXWQAAgrAXWQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiIzYzAwYjJjNy1kNGVjLTRhNjktYWNkYi0xOTBjZGQxYjA2MTAiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJSnZMNE91amJUQnlkc0JLYkl6MjhDVmJBeUJKeTNNTTdqeG9NUHVQRVZZLyJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA") } instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610 instance_runtime = InstanceRuntimeState { propolis_id: Some(1ddde876-90f4-44c8-a71b-91c98435b51a (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-08-23T22:04:04.327434450Z } metadata = InstanceMetadata { silo_id: 84f4ecf9-ccd6-4dde-85c8-c749363b2fc0, project_id: 75873103-03bb-4564-9380-cb8726446850 } propolis_addr = [fd00:1122:3344:101::1:0]:12400 propolis_id = 1ddde876-90f4-44c8-a71b-91c98435b51a vmm_runtime = VmmRuntimeState { state: Starting, gen: Generation(1), time_updated: 2024-08-23T22:04:04.266535Z }
29922024-08-23T22:04:04.783ZINFOSledAgent (InstanceManager): registering new instance file = sled-agent/src/instance_manager.rs:603 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610 (instance)
29932024-08-23T22:04:04.783ZINFOSledAgent (InstanceManager): initializing new Instance file = sled-agent/src/instance.rs:994 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610 migration_id = None propolis_id = 1ddde876-90f4-44c8-a71b-91c98435b51a state = InstanceInitialState { hardware: InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: ab4b9722-63b3-4972-8fe1-3031f72ea0c3, kind: Instance { id: 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 227, 128])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(10829320), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.1.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.1.176), floating_ips: [], firewall_rules: [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: ab4b9722-63b3-4972-8fe1-3031f72ea0c3, kind: Instance { id: 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 227, 128])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(10829320), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: ab4b9722-63b3-4972-8fe1-3031f72ea0c3, kind: Instance { id: 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 227, 128])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(10829320), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(10829320))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: ab4b9722-63b3-4972-8fe1-3031f72ea0c3, kind: Instance { id: 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 227, 128])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(10829320), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-491aa771766", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 5eaa0251-8bc1-48a1-8803-cb58a8bbfe9a, read_only_parent: Some(Volume { block_size: 512, id: 7524761e-59da-402c-a1b9-719097044aad, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 4712ad4e-9156-4dba-9c51-ba0326c965f5, key: Some("jW1c5RNZuQwtt44kNqIs0WM52ryD4P56RmdYsqD+pbI="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::1a]:19002", "[fd00:1122:3344:101::12]:19002", "[fd00:1122:3344:101::15]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 5eaa0251-8bc1-48a1-8803-cb58a8bbfe9a, key: Some("mMBh8eQ+6IHiSbQUR1SypnKTM3vaGv8fE2/Z6KzO1sU="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::13]:19001", "[fd00:1122:3344:101::14]:19000", "[fd00:1122:3344:101::16]:19000"] } }] } }], cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAABJgrAXWRdZAACCsBdZAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAASYKwF1kXWQAAgrAXWQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiIzYzAwYjJjNy1kNGVjLTRhNjktYWNkYi0xOTBjZGQxYjA2MTAiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJSnZMNE91amJUQnlkc0JLYkl6MjhDVmJBeUJKeTNNTTdqeG9NUHVQRVZZLyJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA") }, vmm_runtime: VmmRuntimeState { state: Starting, gen: Generation(1), time_updated: 2024-08-23T22:04:04.266535Z }, propolis_addr: [fd00:1122:3344:101::1:0]:12400, migration_id: None }
29942024-08-23T22:04:04.787ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 15860 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:54383 req_id = db022aa3-ba50-4ddf-aba6-019a71183a0c response_code = 200 uri = /instances/3c00b2c7-d4ec-4a69-acdb-190cdd1b0610
29952024-08-23T22:04:04.924ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:50264
29962024-08-23T22:04:04.930ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 5690 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:50264 req_id = b6470628-dadc-4b76-8def-a0ec9c4297f2 response_code = 200 uri = /instances/3c00b2c7-d4ec-4a69-acdb-190cdd1b0610/state
29972024-08-23T22:04:04.931ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte6", ip: 172.30.0.5, mac: MacAddr6([168, 64, 37, 240, 227, 128]), slot: 0, vni: Vni { inner: 10829320 }, subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.0.1 } }) }
29982024-08-23T22:04:04.932ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
29992024-08-23T22:04:04.971ZINFOSledAgent (InstanceManager): Configuring new Omicron zone: oxz_propolis-server_1ddde876-90f4-44c8-a71b-91c98435b51a file = illumos-utils/src/zone.rs:316 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610
30002024-08-23T22:04:04.994ZINFOSledAgent (InstanceManager): Installing Omicron zone: oxz_propolis-server_1ddde876-90f4-44c8-a71b-91c98435b51a file = illumos-utils/src/zone.rs:354 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610
30012024-08-23T22:04:06.527ZINFOSledAgent (InstanceManager): Profile for oxz_propolis-server_1ddde876-90f4-44c8-a71b-91c98435b51a: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="system/illumos/propolis-server"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlInstance0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1:0'/> <propval type="astring" name="listen_port" value='12400'/> <propval type="astring" name="metric_addr" value='dns'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610
30022024-08-23T22:04:06.527ZINFOSledAgent (InstanceManager): Booting oxz_propolis-server_1ddde876-90f4-44c8-a71b-91c98435b51a zone file = illumos-utils/src/running_zone.rs:488 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610 zone = oxz_propolis-server_1ddde876-90f4-44c8-a71b-91c98435b51a
30032024-08-23T22:04:06.741ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_1ddde876-90f4-44c8-a71b-91c98435b51a") failed: Property not found. retry in 31.986664ms file = illumos-utils/src/svc.rs:36 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610 zone = oxz_propolis-server_1ddde876-90f4-44c8-a71b-91c98435b51a
30042024-08-23T22:04:06.782ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_1ddde876-90f4-44c8-a71b-91c98435b51a") failed: Property not found. retry in 63.609354ms file = illumos-utils/src/svc.rs:36 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610 zone = oxz_propolis-server_1ddde876-90f4-44c8-a71b-91c98435b51a
30052024-08-23T22:04:06.861ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_1ddde876-90f4-44c8-a71b-91c98435b51a") failed: Property not found. retry in 262.020962ms file = illumos-utils/src/svc.rs:36 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610 zone = oxz_propolis-server_1ddde876-90f4-44c8-a71b-91c98435b51a
30062024-08-23T22:04:07.137ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_1ddde876-90f4-44c8-a71b-91c98435b51a") failed: Property not found. retry in 424.740208ms file = illumos-utils/src/svc.rs:36 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610 zone = oxz_propolis-server_1ddde876-90f4-44c8-a71b-91c98435b51a
30072024-08-23T22:04:07.578ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_1ddde876-90f4-44c8-a71b-91c98435b51a") failed: Property not found. retry in 1.104230603s file = illumos-utils/src/svc.rs:36 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610 zone = oxz_propolis-server_1ddde876-90f4-44c8-a71b-91c98435b51a
30082024-08-23T22:04:08.695ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_1ddde876-90f4-44c8-a71b-91c98435b51a") failed: Property not found. retry in 942.298785ms file = illumos-utils/src/svc.rs:36 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610 zone = oxz_propolis-server_1ddde876-90f4-44c8-a71b-91c98435b51a
30092024-08-23T22:04:09.400ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 11584 local_addr = [fd00:1122:3344:101::1]:46625 method = GET remote_addr = [fd00:1122:3344:101::d]:62985 req_id = 672c1e59-fd29-4547-a38b-d8000cbad926 response_code = 200 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be uri = /5050fe14-0845-40b2-872d-d70a5b2b56be
30102024-08-23T22:04:09.646ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_1ddde876-90f4-44c8-a71b-91c98435b51a") failed: Property not found. retry in 623.998319ms file = illumos-utils/src/svc.rs:36 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610 zone = oxz_propolis-server_1ddde876-90f4-44c8-a71b-91c98435b51a
30112024-08-23T22:04:10.282ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_1ddde876-90f4-44c8-a71b-91c98435b51a") failed: Property not found. retry in 555.162154ms file = illumos-utils/src/svc.rs:36 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610 zone = oxz_propolis-server_1ddde876-90f4-44c8-a71b-91c98435b51a
30122024-08-23T22:04:10.849ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_1ddde876-90f4-44c8-a71b-91c98435b51a") failed: Property not found. retry in 1.489972482s file = illumos-utils/src/svc.rs:36 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610 zone = oxz_propolis-server_1ddde876-90f4-44c8-a71b-91c98435b51a
30132024-08-23T22:04:12.370ZINFOSledAgent (InstanceManager): Started propolis in zone: oxz_propolis-server_1ddde876-90f4-44c8-a71b-91c98435b51a file = sled-agent/src/instance.rs:1448 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610
30142024-08-23T22:04:12.380ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_1ddde876-90f4-44c8-a71b-91c98435b51a") failed: Property not found. retry in 61.610866ms file = illumos-utils/src/svc.rs:36 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610
30152024-08-23T22:04:12.450ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_1ddde876-90f4-44c8-a71b-91c98435b51a") failed: Property not found. retry in 122.755011ms file = illumos-utils/src/svc.rs:36 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610
30162024-08-23T22:04:12.582ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_1ddde876-90f4-44c8-a71b-91c98435b51a") failed: Property not found. retry in 107.560314ms file = illumos-utils/src/svc.rs:36 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610
30172024-08-23T22:04:12.700ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_1ddde876-90f4-44c8-a71b-91c98435b51a") failed: Property not found. retry in 207.738962ms file = illumos-utils/src/svc.rs:36 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610
30182024-08-23T22:04:12.919ZINFOSledAgent (InstanceManager): Propolis SMF service is online file = sled-agent/src/instance.rs:1457 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610
30192024-08-23T22:04:12.975ZINFOSledAgent (InstanceManager): Propolis HTTP server online file = sled-agent/src/instance.rs:1481 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610
30202024-08-23T22:04:12.975ZINFOSledAgent (InstanceManager): Sending ensure request to propolis: InstanceEnsureRequest { cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAABJgrAXWRdZAACCsBdZAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAASYKwF1kXWQAAgrAXWQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiIzYzAwYjJjNy1kNGVjLTRhNjktYWNkYi0xOTBjZGQxYjA2MTAiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJSnZMNE91amJUQnlkc0JLYkl6MjhDVmJBeUJKeTNNTTdqeG9NUHVQRVZZLyJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"), disks: [DiskRequest { device: "nvme", name: "disk-491aa771766", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 5eaa0251-8bc1-48a1-8803-cb58a8bbfe9a, read_only_parent: Some(Volume { block_size: 512, id: 7524761e-59da-402c-a1b9-719097044aad, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 4712ad4e-9156-4dba-9c51-ba0326c965f5, key: Some("jW1c5RNZuQwtt44kNqIs0WM52ryD4P56RmdYsqD+pbI="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::1a]:19002", "[fd00:1122:3344:101::12]:19002", "[fd00:1122:3344:101::15]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 5eaa0251-8bc1-48a1-8803-cb58a8bbfe9a, key: Some("mMBh8eQ+6IHiSbQUR1SypnKTM3vaGv8fE2/Z6KzO1sU="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::13]:19001", "[fd00:1122:3344:101::14]:19000", "[fd00:1122:3344:101::16]:19000"] } }] } }], migrate: None, nics: [NetworkInterfaceRequest { name: "opte6", slot: Slot(0) }], properties: InstanceProperties { bootrom_id: 00000000-0000-0000-0000-000000000000, description: "Test description", id: 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610, image_id: 00000000-0000-0000-0000-000000000000, memory: 1024, metadata: InstanceMetadata { project_id: 75873103-03bb-4564-9380-cb8726446850, silo_id: 84f4ecf9-ccd6-4dde-85c8-c749363b2fc0, sled_id: 5050fe14-0845-40b2-872d-d70a5b2b56be, sled_model: "i86pc", sled_revision: 0, sled_serial: "buskin" }, name: "localshark", vcpus: 2 } } file = sled-agent/src/instance.rs:730 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610
30212024-08-23T22:04:13.118ZINFOSledAgent (InstanceManager): result of instance_ensure call is Ok(InstanceEnsureResponse { migrate: None }) file = sled-agent/src/instance.rs:732 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610
30222024-08-23T22:04:13.120ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Creating), migration_in: None, migration_out: None, time: 2024-08-23T22:04:13.120214555Z } file = sled-agent/src/instance.rs:600 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610
30232024-08-23T22:04:13.120ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:622 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610 new_vmm_state = VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-08-23T22:04:13.120214555Z } propolis_id = 1ddde876-90f4-44c8-a71b-91c98435b51a
30242024-08-23T22:04:13.120ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:520 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610 state = SledInstanceState { propolis_id: 1ddde876-90f4-44c8-a71b-91c98435b51a (propolis), vmm_state: VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-08-23T22:04:13.120214555Z }, migration_in: None, migration_out: None }
30252024-08-23T22:04:13.162ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Running), migration_in: None, migration_out: None, time: 2024-08-23T22:04:13.162041703Z } file = sled-agent/src/instance.rs:600 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610
30262024-08-23T22:04:13.162ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:622 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610 new_vmm_state = VmmRuntimeState { state: Running, gen: Generation(3), time_updated: 2024-08-23T22:04:13.162041703Z } propolis_id = 1ddde876-90f4-44c8-a71b-91c98435b51a
30272024-08-23T22:04:13.162ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:520 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610 state = SledInstanceState { propolis_id: 1ddde876-90f4-44c8-a71b-91c98435b51a (propolis), vmm_state: VmmRuntimeState { state: Running, gen: Generation(3), time_updated: 2024-08-23T22:04:13.162041703Z }, migration_in: None, migration_out: None }
30282024-08-23T22:04:26.539ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:36302
30292024-08-23T22:04:26.539ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 58 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:36302 req_id = bd9a4a70-677c-4eac-a0d4-5f3828585600 response_code = 200 uri = /baseboard
30302024-08-23T22:04:30.814ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:58600
30312024-08-23T22:04:30.814ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 69 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:58600 req_id = 6c5d5915-3a65-4b63-9898-fd7c2141fae9 response_code = 200 uri = /vpc-routes
30322024-08-23T22:04:30.859ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 853 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:58600 req_id = ba3ea461-6a71-4d9e-92bb-4a219cf15da4 response_code = 204 uri = /vpc-routes
30332024-08-23T22:04:30.953ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:33572
30342024-08-23T22:04:30.956ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 117 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:33572 req_id = 4e689a84-4740-4439-8448-d5c1cf238da9 response_code = 200 uri = /instances/3c00b2c7-d4ec-4a69-acdb-190cdd1b0610/state
30352024-08-23T22:04:31.079ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:52090
30362024-08-23T22:04:31.080ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 844 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:52090 req_id = 76d9357a-10fd-4895-963b-adf6cea5e756 response_code = 200 uri = /v2p
30372024-08-23T22:04:31.240ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:58253
30382024-08-23T22:04:31.240ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
30392024-08-23T22:04:31.279ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
30402024-08-23T22:04:31.345ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 104987 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:58253 req_id = 499ee77f-ba1d-4bff-95fb-cdf202fcf138 response_code = 204 uri = /switch-ports
30412024-08-23T22:04:31.345ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:43793
30422024-08-23T22:04:31.345ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 212 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:43793 req_id = e2b9ad0c-5d01-4701-8300-df246f391214 response_code = 200 uri = /instances/3c00b2c7-d4ec-4a69-acdb-190cdd1b0610/state
30432024-08-23T22:04:31.346ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1207 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:58253 req_id = 04e125e5-36bc-43e1-9842-44616a79d0db response_code = 200 uri = /network-bootstore-config
30442024-08-23T22:04:31.393ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:48591
30452024-08-23T22:04:31.394ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1145 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:48591 req_id = 3b744075-1d7f-4f26-abc2-5bbce130a773 response_code = 200 uri = /v2p
30462024-08-23T22:04:31.405ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58828
30472024-08-23T22:04:31.405ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 66 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58828 req_id = 39632505-b85c-4419-8a58-34f0d7a779dd response_code = 200 uri = /vpc-routes
30482024-08-23T22:04:31.419ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 143 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:58828 req_id = 9ee725cc-9b7c-43a4-b123-5b7ca0c49d63 response_code = 204 uri = /vpc-routes
30492024-08-23T22:04:31.449ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:56346
30502024-08-23T22:04:31.449ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 108 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:56346 req_id = 3be106a9-23b5-41ba-8d73-1b55673dd740 response_code = 200 uri = /v2p
30512024-08-23T22:04:31.500ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:45397
30522024-08-23T22:04:31.500ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 52 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:45397 req_id = b6d0852e-1656-4f8e-8a03-5fa061009434 response_code = 200 uri = /vpc-routes
30532024-08-23T22:04:31.512ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 90 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:45397 req_id = d4d2098a-da75-4224-91df-106215ff98b4 response_code = 204 uri = /vpc-routes
30542024-08-23T22:04:31.531ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:51788
30552024-08-23T22:04:31.533ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 82 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:51788 req_id = f6b9f601-b724-4fe6-b94d-8ab37fe8757a response_code = 200 uri = /instances/3c00b2c7-d4ec-4a69-acdb-190cdd1b0610/state
30562024-08-23T22:04:31.744ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:43515
30572024-08-23T22:04:31.744ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
30582024-08-23T22:04:31.783ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
30592024-08-23T22:04:31.840ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 95931 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:43515 req_id = dccd3984-891c-4fa3-8c9e-b3150600d5e9 response_code = 204 uri = /switch-ports
30602024-08-23T22:04:31.840ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:43209
30612024-08-23T22:04:31.840ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4033
30622024-08-23T22:04:31.844ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 3649 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:43515 req_id = a1d29a18-b48f-42af-8907-3399cca3fb4b response_code = 200 uri = /network-bootstore-config
30632024-08-23T22:04:31.875ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4057
30642024-08-23T22:04:31.930ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 89434 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:43209 req_id = 6a614e8e-9629-4c1d-8b5f-794d66482be0 response_code = 204 uri = /switch-ports
30652024-08-23T22:04:31.930ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 154 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:43209 req_id = a32e3f2d-83a7-4a0c-90a9-2b42c6af09d0 response_code = 200 uri = /network-bootstore-config
30662024-08-23T22:04:38.594ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59349
30672024-08-23T22:04:38.595ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 527 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:59349 req_id = e573b585-749e-4531-b184-7ac4d5c24e9e response_code = 200 uri = /instances/3c00b2c7-d4ec-4a69-acdb-190cdd1b0610/state
30682024-08-23T22:04:38.595ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Stopping), migration_in: None, migration_out: None, time: 2024-08-23T22:04:38.595241149Z } file = sled-agent/src/instance.rs:600 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610
30692024-08-23T22:04:38.595ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:622 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610 new_vmm_state = VmmRuntimeState { state: Stopping, gen: Generation(5), time_updated: 2024-08-23T22:04:38.595241149Z } propolis_id = 1ddde876-90f4-44c8-a71b-91c98435b51a
30702024-08-23T22:04:38.595ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:520 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610 state = SledInstanceState { propolis_id: 1ddde876-90f4-44c8-a71b-91c98435b51a (propolis), vmm_state: VmmRuntimeState { state: Stopping, gen: Generation(5), time_updated: 2024-08-23T22:04:38.595241149Z }, migration_in: None, migration_out: None }
30712024-08-23T22:04:38.632ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Destroyed), migration_in: None, migration_out: None, time: 2024-08-23T22:04:38.632015802Z } file = sled-agent/src/instance.rs:600 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610
30722024-08-23T22:04:38.632ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:622 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610 new_vmm_state = VmmRuntimeState { state: Destroyed, gen: Generation(6), time_updated: 2024-08-23T22:04:38.632015802Z } propolis_id = 1ddde876-90f4-44c8-a71b-91c98435b51a
30732024-08-23T22:04:38.632ZINFOSledAgent (InstanceManager): terminating VMM that has exited file = sled-agent/src/instance.rs:636 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610
30742024-08-23T22:04:38.632ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
30752024-08-23T22:04:38.632ZWARNSledAgent (kstat-sampler-worker): target removed with queued samples file = oximeter/instruments/src/kstat/sampler.rs:557 id = 12316762901450333873 n_samples = 12 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
30762024-08-23T22:04:38.632ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:406
30772024-08-23T22:04:38.632ZINFOSledAgent (ZoneBundler): creating zone bundle context = ZoneBundleContext { storage_dirs: ["/pool/int/156808b0-3956-455e-a8e1-2217e0afca54/debug/bundle/zone", "/pool/int/fe7217f3-1112-4ad5-a4fc-b58965332aab/debug/bundle/zone"], cause: TerminatedInstance, extra_log_dirs: ["/pool/ext/c4828395-89b1-4653-93a7-55b242871c48/crypt/debug", "/pool/ext/c4fe4e01-40de-4c9e-857a-e6870bd84153/crypt/debug", "/pool/ext/0bb3d35c-cbec-445b-9b3a-58337934f8de/crypt/debug", "/pool/ext/44b92f88-0c07-4732-953e-9cac63715495/crypt/debug", "/pool/ext/44ff68f6-909f-4df8-8e90-d68ca84323b3/crypt/debug", "/pool/ext/dad70eae-b83f-4c3a-a32e-2a6a8946803c/crypt/debug", "/pool/ext/ec50006e-a21c-4820-b439-48f6d6b60c97/crypt/debug", "/pool/ext/e1a016b1-16ad-48d4-a26f-2e916f51bc80/crypt/debug", "/pool/ext/7ac03450-9a27-4ec4-9669-9eb798da4419/crypt/debug"] } file = sled-agent/src/zone_bundle.rs:339 zone_name = oxz_propolis-server_1ddde876-90f4-44c8-a71b-91c98435b51a
30782024-08-23T22:04:38.632ZWARNSledAgent (kstat-sampler-worker): target removed with queued samples file = oximeter/instruments/src/kstat/sampler.rs:557 id = 13157610138689833664 n_samples = 12 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be
30792024-08-23T22:04:39.254ZINFOSledAgent (ZoneBundler): finished zone bundle file = sled-agent/src/zone_bundle.rs:1137 metadata = ZoneBundleMetadata { id: ZoneBundleId { zone_name: "oxz_propolis-server_1ddde876-90f4-44c8-a71b-91c98435b51a", bundle_id: 004b52aa-c9b0-4370-ab42-1a7ca67e68c9 }, time_created: 2024-08-23T22:04:38.633119591Z, version: 0, cause: TerminatedInstance }
30802024-08-23T22:04:39.254ZWARNSledAgent (InstanceManager): Halting and removing zone: oxz_propolis-server_1ddde876-90f4-44c8-a71b-91c98435b51a file = sled-agent/src/instance.rs:831 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610
30812024-08-23T22:04:39.395ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7598 local_addr = [fd00:1122:3344:101::1]:46625 method = GET remote_addr = [fd00:1122:3344:101::d]:62985 req_id = ddb66766-aa50-46c0-9214-1f5c29ea5db4 response_code = 200 sled_id = 5050fe14-0845-40b2-872d-d70a5b2b56be uri = /5050fe14-0845-40b2-872d-d70a5b2b56be
30822024-08-23T22:04:41.005ZINFOSledAgent (InstanceManager): halt_and_remove_logged: Previous zone state: Running file = illumos-utils/src/zone.rs:270 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610
30832024-08-23T22:04:41.021ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:904 id = RouterId { vni: Vni(10829320), kind: System }
30842024-08-23T22:04:41.021ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:904 id = RouterId { vni: Vni(10829320), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }
30852024-08-23T22:04:41.022ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:520 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610 state = SledInstanceState { propolis_id: 1ddde876-90f4-44c8-a71b-91c98435b51a (propolis), vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-08-23T22:04:41.022105498Z }, migration_in: None, migration_out: None }
30862024-08-23T22:04:41.049ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:520 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610 state = SledInstanceState { propolis_id: 1ddde876-90f4-44c8-a71b-91c98435b51a (propolis), vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-08-23T22:04:41.022105498Z }, migration_in: None, migration_out: None }
30872024-08-23T22:04:41.054ZINFOSledAgent (InstanceManager): State monitoring task complete file = sled-agent/src/instance.rs:768 instance_id = 3c00b2c7-d4ec-4a69-acdb-190cdd1b0610
30882024-08-23T22:04:41.434ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:44729
30892024-08-23T22:04:41.434ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:44729 req_id = 898a3775-2671-4f27-ae98-48b5085d4d83 response_code = 200 uri = /vpc-routes
30902024-08-23T22:04:41.439ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 126 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:44729 req_id = 01c35d97-cfe7-48e1-b9eb-c6b3b5ae3a54 response_code = 204 uri = /vpc-routes
30912024-08-23T22:04:41.452ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:50439
30922024-08-23T22:04:41.452ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 664 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:50439 req_id = ab79807d-de0c-4232-b3e9-6ca924f3d385 response_code = 200 uri = /v2p
30932024-08-23T22:04:41.730ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36468
30942024-08-23T22:04:41.730ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 152 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:36468 req_id = 1a27c674-443b-46ba-94c4-2d56898a3876 response_code = 200 uri = /v2p
30952024-08-23T22:04:42.764ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63700
30962024-08-23T22:04:42.764ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 106 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:63700 req_id = 69975a7f-2853-4227-86b0-b58a957c446f response_code = 200 uri = /vpc-routes
30972024-08-23T22:04:42.770ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:63700 req_id = c4d3fe51-73fc-4a4d-a96f-3c739708254a response_code = 204 uri = /vpc-routes