Max level shown:
1 [ Jul 26 02:33:54 Enabled. ]
2 [ Jul 26 02:33:54 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]
3 [ Jul 26 02:33:54 Method "start" exited with status 0. ]
4 [ Jul 26 02:33:54 Rereading configuration. ]
5 [ Jul 26 02:33:54 No 'refresh' method defined. Treating as :true. ]
6 note: configured to log to "/dev/stdout"
72024-07-26T02:33:54.136ZINFOSledAgent: Ensuring contract reaper thread file = illumos-utils/src/running_zone.rs:113
82024-07-26T02:33:54.136ZINFOSledAgent (BootstrapAgentStartup): finding links ["net0", "net1"] file = sled-agent/src/bootstrap/pre_server.rs:255
92024-07-26T02:33:54.190ZINFOSledAgent (BootstrapAgentStartup): Starting mg-ddm service addr-objs = [AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }] file = sled-agent/src/bootstrap/pre_server.rs:262
102024-07-26T02:33:54.190ZINFOSledAgent (BootstrapAgentStartup): Importing mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:42 path = /opt/oxide/mg-ddm/pkg/ddm/manifest.xml
112024-07-26T02:33:54.305ZINFOSledAgent (BootstrapAgentStartup): Setting mg-ddm interfaces file = sled-agent/src/bootstrap/maghemite.rs:50 interfaces = ("net0/ll" "net1/ll")
122024-07-26T02:33:54.314ZINFOSledAgent (BootstrapAgentStartup): Enabling mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:56
132024-07-26T02:33:54.322ZINFOSledAgent (BootstrapAgentStartup): Ensuring zfs key directory exists file = sled-agent/src/bootstrap/pre_server.rs:272 path = /var/run/oxide/
142024-07-26T02:33:54.927ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fdb0:18c0:4d0d:a02a::, width: 64 }
152024-07-26T02:33:55.069ZINFOSledAgent: Starting KeyManager file = sled-agent/src/long_running_tasks.rs:125
162024-07-26T02:33:55.069ZINFOSledAgent: Starting StorageManager file = sled-agent/src/long_running_tasks.rs:137
172024-07-26T02:33:55.069ZINFOSledAgent: Starting StorageMonitor file = sled-agent/src/long_running_tasks.rs:150
182024-07-26T02:33:55.069ZINFOSledAgent: Starting HardwareManager file = sled-agent/src/long_running_tasks.rs:172 nongimlet_observed_disks = [] sled_mode = Scrimlet { asic: SoftNpuZone }
192024-07-26T02:33:55.069ZINFOSledAgent (DumpSetup-worker): DumpSetup poll loop started. file = sled-agent/src/dump_setup.rs:520
202024-07-26T02:33:55.069ZINFOSledAgent (HardwareManager): Creating HardwareManager file = sled-hardware/src/illumos/mod.rs:716
212024-07-26T02:33:55.072ZINFOSledAgent (HardwareManager): Generated i86pc baseboard Pc { identifier: "sock", model: "i86pc" } file = sled-hardware/src/illumos/mod.rs:623
222024-07-26T02:33:55.072ZWARNSledAgent (HardwareManager): Device is not a Gimlet (i86pc), proceeding with null hardware view file = sled-hardware/src/illumos/mod.rs:765
232024-07-26T02:33:55.072ZINFOSledAgent: Starting HardwareMonitor file = sled-agent/src/long_running_tasks.rs:186
242024-07-26T02:33:55.072ZINFOSledAgent (HardwareManager): Monitoring for hardware updates file = sled-hardware/src/illumos/mod.rs:824
252024-07-26T02:33:55.072ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/m2_0.vdev
262024-07-26T02:33:55.072ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = None
272024-07-26T02:33:55.072ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:405
282024-07-26T02:33:55.072ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
292024-07-26T02:33:55.072ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } path = /scratch/m2_0.vdev
302024-07-26T02:33:55.084ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_0.vdev file = sled-hardware/src/disk.rs:360
312024-07-26T02:33:55.084ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:371 id = 2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1 (zpool)
322024-07-26T02:33:55.184ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:304 zpool = ZpoolName { id: 2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1 (zpool), kind: Internal }
332024-07-26T02:33:55.398ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:405
342024-07-26T02:33:55.398ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/m2_1.vdev
352024-07-26T02:33:55.398ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
362024-07-26T02:33:55.398ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
372024-07-26T02:33:55.398ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:405
382024-07-26T02:33:55.398ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
392024-07-26T02:33:55.398ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } path = /scratch/m2_1.vdev
402024-07-26T02:33:55.407ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_1.vdev file = sled-hardware/src/disk.rs:360
412024-07-26T02:33:55.407ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:371 id = f09c5312-7869-4743-8a0d-863eb3bf0156 (zpool)
422024-07-26T02:33:55.502ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:304 zpool = ZpoolName { id: f09c5312-7869-4743-8a0d-863eb3bf0156 (zpool), kind: Internal }
432024-07-26T02:33:55.711ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_0.vdev
442024-07-26T02:33:55.711ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f09c5312-7869-4743-8a0d-863eb3bf0156 (zpool), kind: Internal } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
452024-07-26T02:33:55.711ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
462024-07-26T02:33:55.711ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
472024-07-26T02:33:55.711ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:405
482024-07-26T02:33:55.711ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
492024-07-26T02:33:55.711ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_1.vdev
502024-07-26T02:33:55.711ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:405
512024-07-26T02:33:55.711ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f09c5312-7869-4743-8a0d-863eb3bf0156 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
522024-07-26T02:33:55.711ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
532024-07-26T02:33:55.711ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
542024-07-26T02:33:55.711ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
552024-07-26T02:33:55.711ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_2.vdev
562024-07-26T02:33:55.711ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:405
572024-07-26T02:33:55.711ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
582024-07-26T02:33:55.711ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f09c5312-7869-4743-8a0d-863eb3bf0156 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
592024-07-26T02:33:55.711ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_3.vdev
602024-07-26T02:33:55.711ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
612024-07-26T02:33:55.711ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
622024-07-26T02:33:55.711ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f09c5312-7869-4743-8a0d-863eb3bf0156 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
632024-07-26T02:33:55.711ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
642024-07-26T02:33:55.711ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
652024-07-26T02:33:55.711ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:405
662024-07-26T02:33:55.711ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
672024-07-26T02:33:55.711ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_4.vdev
682024-07-26T02:33:55.711ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:405
692024-07-26T02:33:55.711ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
702024-07-26T02:33:55.711ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_5.vdev
712024-07-26T02:33:55.711ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f09c5312-7869-4743-8a0d-863eb3bf0156 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
722024-07-26T02:33:55.711ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:405
732024-07-26T02:33:55.711ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
742024-07-26T02:33:55.711ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
752024-07-26T02:33:55.711ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
762024-07-26T02:33:55.711ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_6.vdev
772024-07-26T02:33:55.711ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:405
782024-07-26T02:33:55.711ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
792024-07-26T02:33:55.711ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f09c5312-7869-4743-8a0d-863eb3bf0156 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
802024-07-26T02:33:55.711ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
812024-07-26T02:33:55.711ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
822024-07-26T02:33:55.711ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_7.vdev
832024-07-26T02:33:55.711ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:405
842024-07-26T02:33:55.711ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
852024-07-26T02:33:55.711ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_8.vdev
862024-07-26T02:33:55.711ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f09c5312-7869-4743-8a0d-863eb3bf0156 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
872024-07-26T02:33:55.711ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
882024-07-26T02:33:55.711ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
892024-07-26T02:33:55.711ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:405
902024-07-26T02:33:55.711ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
912024-07-26T02:33:55.711ZINFOSledAgent: Waiting for boot disk file = sled-agent/src/long_running_tasks.rs:96
922024-07-26T02:33:55.711ZINFOSledAgent: Found boot disk DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-agent/src/long_running_tasks.rs:98
932024-07-26T02:33:55.711ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f09c5312-7869-4743-8a0d-863eb3bf0156 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
942024-07-26T02:33:55.711ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
952024-07-26T02:33:55.711ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
962024-07-26T02:33:55.711ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
972024-07-26T02:33:55.711ZINFOSledAgent: Starting Bootstore file = sled-agent/src/long_running_tasks.rs:210
982024-07-26T02:33:55.711ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f09c5312-7869-4743-8a0d-863eb3bf0156 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
992024-07-26T02:33:55.711ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
1002024-07-26T02:33:55.711ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
1012024-07-26T02:33:55.711ZINFOSledAgent (bootstore): No ledger in /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1022024-07-26T02:33:55.711ZINFOSledAgent (bootstore): No ledger in /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1032024-07-26T02:33:55.711ZINFOSledAgent (bootstore): No ledger found. Loading Fsm::State as Uninitialized file = bootstore/src/schemes/v0/storage.rs:83 peer_id = pc-sock-i86pc
1042024-07-26T02:33:55.711ZINFOSledAgent (bootstore): No ledger in /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1052024-07-26T02:33:55.711ZINFOSledAgent (bootstore): No ledger in /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1062024-07-26T02:33:55.711ZINFOSledAgent (bootstore): No ledger found for network config file = bootstore/src/schemes/v0/storage.rs:146 peer_id = pc-sock-i86pc
1072024-07-26T02:33:55.711ZINFOSledAgent: Starting Bootstore DDMD poller file = sled-agent/src/long_running_tasks.rs:215
1082024-07-26T02:33:55.711ZINFOSledAgent: Starting ZoneBundler related tasks file = sled-agent/src/long_running_tasks.rs:230
1092024-07-26T02:33:55.722ZINFOSledAgent (ServiceManager): Creating ServiceManager file = sled-agent/src/services.rs:722
1102024-07-26T02:33:55.722ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:292 next_cleanup = Instant { tv_sec: 12794, tv_nsec: 811918527 } time_to_next_cleanup = 599.999923148s
1112024-07-26T02:33:55.722ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3661
1122024-07-26T02:33:55.722ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
1132024-07-26T02:33:55.722ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/sled-agent-request.json file = common/src/ledger.rs:188
1142024-07-26T02:33:55.722ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/sled-agent-request.json file = common/src/ledger.rs:188
1152024-07-26T02:33:55.725ZINFOSledAgent (dropshot (BootstrapAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:205 local_addr = [fdb0:18c0:4d0d:a02a::1]:80
1162024-07-26T02:33:55.725ZINFOSledAgent (SledAgentSprocketsServer): Started listening file = sled-agent/src/bootstrap/sprockets_server.rs:44 local_addr = [fdb0:18c0:4d0d:a02a::1]:12346
1172024-07-26T02:33:55.725ZINFOSledAgent (RSS): Injecting RSS configuration: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.2.164, last: 10.151.2.173, }, ), ], external_dns_ips: [ 10.151.2.164, 10.151.2.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.2.100, vlan_id: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.2.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, } file = sled-agent/src/rack_setup/service.rs:1017
1182024-07-26T02:33:55.725ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
1192024-07-26T02:33:55.725ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
1202024-07-26T02:33:55.725ZINFOSledAgent (RSS): No ledger in /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1212024-07-26T02:33:55.725ZINFOSledAgent (RSS): No ledger in /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1222024-07-26T02:33:55.725ZINFOSledAgent (RSS): RSS configuration has not been fully applied yet file = sled-agent/src/rack_setup/service.rs:1077
1232024-07-26T02:33:55.725ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
1242024-07-26T02:33:55.725ZINFOSledAgent (RSS): No ledger in /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/rss-sled-plan.json file = common/src/ledger.rs:188
1252024-07-26T02:33:55.725ZINFOSledAgent (RSS): No ledger in /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/rss-sled-plan.json file = common/src/ledger.rs:188
1262024-07-26T02:33:55.725ZINFOSledAgent (RSS): Creating new allocation plan file = sled-agent/src/rack_setup/service.rs:1118
1272024-07-26T02:33:55.725ZINFOSledAgent (RSS): Serializing plan file = sled-agent/src/rack_setup/plan/sled.rs:156
1282024-07-26T02:33:55.725ZINFOSledAgent (RSS): Creating plan for the sled at fdb0:18c0:4d0d:a02a::1 file = sled-agent/src/rack_setup/plan/sled.rs:129
1292024-07-26T02:33:55.726ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
1302024-07-26T02:33:55.726ZINFOSledAgent (RSS): Writing ledger to /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1312024-07-26T02:33:55.726ZINFOSledAgent (RSS): Writing ledger to /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1322024-07-26T02:33:55.726ZINFOSledAgent (RSS): Sled plan written to storage file = sled-agent/src/rack_setup/plan/sled.rs:176
1332024-07-26T02:33:55.726ZINFOSledAgent (RSS): Writing Rack Network Configuration to bootstore file = sled-agent/src/rack_setup/service.rs:1151
1342024-07-26T02:33:55.726ZINFOSledAgent (bootstore): Attempting to update network config with generation: 1, current_generation: 0 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-sock-i86pc
1352024-07-26T02:33:55.726ZINFOSledAgent (bootstore): Writing ledger to /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
1362024-07-26T02:33:55.726ZINFOSledAgent (bootstore): Writing ledger to /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
1372024-07-26T02:33:55.727ZINFOSledAgent (bootstore): Broadcasting network config with generation 1 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-sock-i86pc
1382024-07-26T02:33:55.727ZINFOSledAgent (BootstrapAgentRssHandler): Received initialization request from RSS file = sled-agent/src/bootstrap/rss_handle.rs:217 request = StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: ea0e0fc4-fc93-4de6-b687-98567d824e83, rack_id: 6101b963-ee1b-430c-ad60-5a5922d708b7, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } target_sled = [fdb0:18c0:4d0d:a02a::1]:12346
1392024-07-26T02:33:55.727ZINFOSledAgent (SledAgentSprocketsServer): Accepted connection file = sled-agent/src/bootstrap/sprockets_server.rs:66 remote_addr = [fdb0:18c0:4d0d:a02a::1]:55089
1402024-07-26T02:33:55.727ZINFOSledAgent (SledAgentMain): Loading Sled Agent: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: ea0e0fc4-fc93-4de6-b687-98567d824e83, rack_id: 6101b963-ee1b-430c-ad60-5a5922d708b7, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } file = sled-agent/src/bootstrap/server.rs:359
1412024-07-26T02:33:55.727ZINFOSledAgent (SledAgentMain): KeyManager: using hardcoded secret retriever file = sled-agent/src/bootstrap/server.rs:374
1422024-07-26T02:33:55.727ZINFOSledAgent (StorageManager): Received KeyManagerReady file = sled-storage/src/manager.rs:405
1432024-07-26T02:33:55.727ZINFOSledAgent (StorageManager): Loading storage config file = sled-storage/src/manager.rs:573
1442024-07-26T02:33:55.727ZINFOSledAgent: setting up sled agent server file = sled-agent/src/server.rs:44
1452024-07-26T02:33:55.727ZINFOSledAgent (StorageManager): No ledger in /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_ledger
1462024-07-26T02:33:55.727ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:101::, width: 64 }
1472024-07-26T02:33:55.727ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
1482024-07-26T02:33:55.727ZINFOSledAgent (StorageManager): No ledger in /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_ledger
1492024-07-26T02:33:55.727ZINFOSledAgent (StorageManager): No ledger of physical disks exists file = sled-storage/src/manager.rs:562
1502024-07-26T02:33:55.727ZINFOSledAgent (StorageManager): KeyManager ready, but no ledger detected file = sled-storage/src/manager.rs:590
1512024-07-26T02:33:55.727ZINFOSledAgent (StorageManager): automatically managing disks file = sled-storage/src/manager.rs:394
1522024-07-26T02:33:55.727ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:374
1532024-07-26T02:33:55.727ZINFOSledAgent (StorageManager): Successfully synchronized disks without error file = sled-storage/src/manager.rs:480 result = DisksManagementResult { status: [] }
1542024-07-26T02:33:55.729ZINFOSledAgent (ServiceManager): Enabling switch zone (new) file = sled-agent/src/services.rs:3895
1552024-07-26T02:33:55.729ZINFOSledAgent (ServiceManager): Starting switch zone file = sled-agent/src/services.rs:4268
1562024-07-26T02:33:55.760ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
1572024-07-26T02:33:55.781ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:316
1582024-07-26T02:33:55.798ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:354
1592024-07-26T02:33:55.845ZINFOSledAgent: SledAgent::new(..) starting file = sled-agent/src/sled_agent.rs:368 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
1602024-07-26T02:33:55.845ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
1612024-07-26T02:33:55.845ZINFOSledAgent: Requested swap device of size 64 GiB file = sled-agent/src/sled_agent.rs:380 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
1622024-07-26T02:33:55.845ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1632024-07-26T02:33:55.849ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 134.737258ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "96da0c5b-a072-4adf-b782-dcacd1638aad", "content-length": "124", "date": "Fri, 26 Jul 2024 02:33:54 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "96da0c5b-a072-4adf-b782-dcacd1638aad" } file = clients/ddm-admin-client/src/lib.rs:177
1642024-07-26T02:33:55.859ZINFOSledAgent: attempting to create encrypted zvol: name="oxi_2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/swap", size_gb=64 file = sled-agent/src/swap_device.rs:153
1652024-07-26T02:33:55.883ZINFOSledAgent: successfully created encrypted zvol: name="oxi_2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/swap", size_gb=64 file = sled-agent/src/swap_device.rs:249
1662024-07-26T02:33:55.883ZINFOSledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/swap" file = sled-agent/src/swap_device.rs:96
1672024-07-26T02:33:55.890ZINFOSledAgent: Mounting backing filesystems file = sled-agent/src/sled_agent.rs:395 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
1682024-07-26T02:33:55.890ZINFOSledAgent (BackingFs): Processing fmd file = sled-agent/src/backing_fs.rs:126
1692024-07-26T02:33:55.890ZINFOSledAgent (BackingFs): Ensuring dataset oxi_2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/backing/fmd file = sled-agent/src/backing_fs.rs:136
1702024-07-26T02:33:55.963ZINFOSledAgent (BackingFs): Stopping service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:170
1712024-07-26T02:33:55.984ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1722024-07-26T02:33:55.985ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 90.877879ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "16eaa63a-1367-494e-840e-6fb2c88dc57f", "content-length": "124", "date": "Fri, 26 Jul 2024 02:33:55 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "16eaa63a-1367-494e-840e-6fb2c88dc57f" } file = clients/ddm-admin-client/src/lib.rs:177
1732024-07-26T02:33:56.077ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1742024-07-26T02:33:56.077ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 73.713032ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "7f01bd36-8f22-4bbb-8b63-7d50c0361984", "content-length": "124", "date": "Fri, 26 Jul 2024 02:33:55 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "7f01bd36-8f22-4bbb-8b63-7d50c0361984" } file = clients/ddm-admin-client/src/lib.rs:177
1752024-07-26T02:33:56.153ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1762024-07-26T02:33:56.153ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 252.194671ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "69b17e8b-cc59-457f-9f39-872ab32ae78b", "content-length": "124", "date": "Fri, 26 Jul 2024 02:33:55 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "69b17e8b-cc59-457f-9f39-872ab32ae78b" } file = clients/ddm-admin-client/src/lib.rs:177
1772024-07-26T02:33:56.183ZINFOSledAgent (BackingFs): Mounting oxi_2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/backing/fmd on /var/fm/fmd file = sled-agent/src/backing_fs.rs:178
1782024-07-26T02:33:56.196ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/rsrc file = sled-agent/src/backing_fs.rs:186
1792024-07-26T02:33:56.196ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/ckpt file = sled-agent/src/backing_fs.rs:186
1802024-07-26T02:33:56.196ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/xprt file = sled-agent/src/backing_fs.rs:186
1812024-07-26T02:33:56.196ZINFOSledAgent (BackingFs): Starting service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:192
1822024-07-26T02:33:56.407ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1832024-07-26T02:33:56.408ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 223.787058ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "5dee47de-f41c-42fe-8135-27bbfb42e2d9", "content-length": "124", "date": "Fri, 26 Jul 2024 02:33:55 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "5dee47de-f41c-42fe-8135-27bbfb42e2d9" } file = clients/ddm-admin-client/src/lib.rs:177
1842024-07-26T02:33:56.633ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1852024-07-26T02:33:56.634ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 212.675498ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "76ed8039-551b-49ac-a0a2-aaeafb8669d6", "content-length": "124", "date": "Fri, 26 Jul 2024 02:33:55 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "76ed8039-551b-49ac-a0a2-aaeafb8669d6" } file = clients/ddm-admin-client/src/lib.rs:177
1862024-07-26T02:33:56.848ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1872024-07-26T02:33:56.848ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 420.906112ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "ef5d251e-3d1d-49f4-826a-a7ac746f6946", "content-length": "124", "date": "Fri, 26 Jul 2024 02:33:55 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "ef5d251e-3d1d-49f4-826a-a7ac746f6946" } file = clients/ddm-admin-client/src/lib.rs:177
1882024-07-26T02:33:56.921ZINFOSledAgent: using '[AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }]' as data links for xde driver file = illumos-utils/src/opte/illumos.rs:90 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
1892024-07-26T02:33:56.922ZINFOSledAgent (dropshot): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:205 local_addr = [fd00:1122:3344:101::1]:33355 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
1902024-07-26T02:33:56.922ZINFOSledAgent (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
1912024-07-26T02:33:56.922ZINFOSledAgent (producer-server): starting oximeter metric producer server address = [fd00:1122:3344:101::1]:33355 file = oximeter/producer/src/lib.rs:283 interval = 30s producer_id = ea0e0fc4-fc93-4de6-b687-98567d824e83 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
1922024-07-26T02:33:56.923ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 157.085485ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
1932024-07-26T02:33:56.968ZINFOSledAgent (VmmReservoirManager): 50% of 102952026112 physical ram = 51476013056 bytes) file = sled-agent/src/vmm_reservoir.rs:267 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
1942024-07-26T02:33:56.968ZINFOSledAgent (VmmReservoirManager): Setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:275 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
1952024-07-26T02:33:56.968ZINFOSledAgent (ServiceManager): sled agent started file = sled-agent/src/services.rs:905 underlay_address = fd00:1122:3344:101::1
1962024-07-26T02:33:56.968ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
1972024-07-26T02:33:56.969ZINFOSledAgent (dropshot (SledAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:205 local_addr = [fd00:1122:3344:101::1]:12345
1982024-07-26T02:33:56.969ZINFOSledAgent (SledAgentMain): Sled Agent loaded; recording configuration file = sled-agent/src/bootstrap/server.rs:429
1992024-07-26T02:33:56.969ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
2002024-07-26T02:33:56.969ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2012024-07-26T02:33:56.969ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2022024-07-26T02:33:56.969ZINFOSledAgent (HardwareMonitor): Sled Agent Started file = sled-agent/src/hardware_monitor.rs:138
2032024-07-26T02:33:56.969ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = Some((fd00:1122:3344:101::2, Some(RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56 }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [PortConfigV2 { routes: [RouteConfig { destination: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), nexthop: 10.151.2.100, vlan_id: None }], addresses: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false }], bgp: [], bfd: [] })))
2042024-07-26T02:33:56.969ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3661
2052024-07-26T02:33:56.969ZINFOSledAgent (SledAgentSprocketsServer): Connection closed file = sled-agent/src/bootstrap/sprockets_server.rs:73 remote_addr = [fdb0:18c0:4d0d:a02a::1]:55089
2062024-07-26T02:33:57.137ZINFOSledAgent (BootstrapAgentRssHandler): Peer agent initialized file = sled-agent/src/bootstrap/rss_handle.rs:108 peer = [fdb0:18c0:4d0d:a02a::1]:12346
2072024-07-26T02:33:57.137ZINFOSledAgent (BootstrapAgentRssHandler): Initialized sled agent file = sled-agent/src/bootstrap/rss_handle.rs:232 target_sled = [fdb0:18c0:4d0d:a02a::1]:12346
2082024-07-26T02:33:57.137ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
2092024-07-26T02:33:57.138ZINFOSledAgent (RSS): No ledger in /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/rss-service-plan-v3.json file = common/src/ledger.rs:188
2102024-07-26T02:33:57.138ZINFOSledAgent (RSS): No ledger in /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/rss-service-plan-v3.json file = common/src/ledger.rs:188
2112024-07-26T02:33:57.138ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
2122024-07-26T02:33:57.138ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
2132024-07-26T02:33:57.138ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 465.983532ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
2142024-07-26T02:33:57.139ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/ea0e0fc4-fc93-4de6-b687-98567d824e83): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:600 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
2152024-07-26T02:33:57.178ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:34661
2162024-07-26T02:33:57.178ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
2172024-07-26T02:33:57.178ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 143 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:34661 req_id = 255c2327-a22c-4074-a20c-36482ac831f9 response_code = 200 uri = /inventory
2182024-07-26T02:33:57.209ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42948
2192024-07-26T02:33:57.209ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 25 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:42948 req_id = d79449f5-1737-4f0c-8e7b-1ae4e1b6f07b response_code = 200 uri = /sled-role
2202024-07-26T02:33:57.211ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
2212024-07-26T02:33:57.211ZINFOSledAgent (RSS): Writing ledger to /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/.rss-service-plan-v3.json.tmp file = common/src/ledger.rs:199
2222024-07-26T02:33:57.211ZINFOSledAgent (RSS): Writing ledger to /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/.rss-service-plan-v3.json.tmp file = common/src/ledger.rs:199
2232024-07-26T02:33:57.211ZINFOSledAgent (RSS): Service plan written to storage file = sled-agent/src/rack_setup/plan/service.rs:830
2242024-07-26T02:33:57.240ZINFOSledAgent (RSS): attempting to set up sled's storage: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: 11f51cfd-88b7-4f18-8951-7d757e18895b, pool_id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: ce0e8de8-ec3e-4a65-9b12-4b3d05e2af7c, pool_id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: 9426f29b-4ad6-432a-9152-3ddfd0b60d09, pool_id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: 6132d37c-1434-4fed-a5c9-ade469b740a0, pool_id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: 6c848580-6df8-44cd-b026-f3c062e46850, pool_id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: 88274dc3-ce1f-4b2a-9966-f91952efd323, pool_id: cb102b97-0352-4c03-8d37-4c80c4475510 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: e10e4f5f-4a5f-47ac-ad38-15eb242bc36f, pool_id: c5566b16-9a9f-498b-a4b1-83ac43308db5 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: 3b3f1b92-8896-4279-9dde-6984c541854b, pool_id: 136ed548-ab63-44da-a187-0fa13dc111a0 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: 20f5fcb2-ea2f-4860-9d58-9d8a18d63e8b, pool_id: 3ed9ecd9-4aeb-4ec2-8876-b9c8ed7df016 (zpool) }] } file = sled-agent/src/rack_setup/service.rs:355 sled_address = [fd00:1122:3344:101::1]:12345
2252024-07-26T02:33:57.240ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:44735
2262024-07-26T02:33:57.241ZINFOSledAgent: physical disks ensure file = sled-agent/src/sled_agent.rs:827 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
2272024-07-26T02:33:57.241ZINFOSledAgent (StorageManager): Received OmicronPhysicalDisksEnsure { config: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: 11f51cfd-88b7-4f18-8951-7d757e18895b, pool_id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: ce0e8de8-ec3e-4a65-9b12-4b3d05e2af7c, pool_id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: 9426f29b-4ad6-432a-9152-3ddfd0b60d09, pool_id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: 6132d37c-1434-4fed-a5c9-ade469b740a0, pool_id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: 6c848580-6df8-44cd-b026-f3c062e46850, pool_id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: 88274dc3-ce1f-4b2a-9966-f91952efd323, pool_id: cb102b97-0352-4c03-8d37-4c80c4475510 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: e10e4f5f-4a5f-47ac-ad38-15eb242bc36f, pool_id: c5566b16-9a9f-498b-a4b1-83ac43308db5 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: 3b3f1b92-8896-4279-9dde-6984c541854b, pool_id: 136ed548-ab63-44da-a187-0fa13dc111a0 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: 20f5fcb2-ea2f-4860-9d58-9d8a18d63e8b, pool_id: 3ed9ecd9-4aeb-4ec2-8876-b9c8ed7df016 (zpool) }] }, tx: ... } file = sled-storage/src/manager.rs:405
2282024-07-26T02:33:57.241ZINFOSledAgent (StorageManager): No ledger in /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2292024-07-26T02:33:57.241ZINFOSledAgent (StorageManager): No ledger in /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2302024-07-26T02:33:57.241ZINFOSledAgent (StorageManager): No previously-stored 'requested disks', creating new ledger file = sled-storage/src/manager.rs:644 request = omicron_physical_disks_ensure
2312024-07-26T02:33:57.241ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:374
2322024-07-26T02:33:57.241ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:414
2332024-07-26T02:33:57.241ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
2342024-07-26T02:33:57.241ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } path = /scratch/u2_0.vdev
2352024-07-26T02:33:57.321ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
2362024-07-26T02:33:57.330ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_0.vdev file = sled-hardware/src/disk.rs:360
2372024-07-26T02:33:57.330ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool)
2382024-07-26T02:33:57.711ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 620.557678ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
2392024-07-26T02:33:57.970ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/ea0e0fc4-fc93-4de6-b687-98567d824e83): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
2402024-07-26T02:33:58.317ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:304 zpool = ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }
2412024-07-26T02:33:58.478ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 2.916927833s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
2422024-07-26T02:33:58.484ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:348
2432024-07-26T02:33:58.484ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:350
2442024-07-26T02:33:58.484ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:354
2452024-07-26T02:33:58.484ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:356
2462024-07-26T02:33:58.484ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2472024-07-26T02:33:58.484ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2482024-07-26T02:33:58.484ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:368
2492024-07-26T02:33:59.021ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/ea0e0fc4-fc93-4de6-b687-98567d824e83): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
2502024-07-26T02:33:59.021ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/ea0e0fc4-fc93-4de6-b687-98567d824e83): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:600 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
2512024-07-26T02:33:59.023ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2522024-07-26T02:33:59.209ZINFOSledAgent (ServiceManager): Setting up dendrite service file = sled-agent/src/services.rs:2472
2532024-07-26T02:33:59.209ZINFOSledAgent (ServiceManager): Setting up lldpd service file = sled-agent/src/services.rs:2690
2542024-07-26T02:33:59.209ZINFOSledAgent (ServiceManager): Setting up MGS service file = sled-agent/src/services.rs:2342
2552024-07-26T02:33:59.209ZINFOSledAgent (ServiceManager): Setting up wicketd service file = sled-agent/src/services.rs:2389
2562024-07-26T02:33:59.209ZINFOSledAgent (ServiceManager): Setting up mgd service file = sled-agent/src/services.rs:2771
2572024-07-26T02:33:59.209ZINFOSledAgent (ServiceManager): Setting up mg-ddm service file = sled-agent/src/services.rs:2813
2582024-07-26T02:33:59.212ZINFOSledAgent (ServiceManager): Setting up tfport service file = sled-agent/src/services.rs:2615
2592024-07-26T02:33:59.214ZINFOSledAgent (ServiceManager): Setting up Simulated SP service file = sled-agent/src/services.rs:2383
2602024-07-26T02:33:59.214ZINFOSledAgent (ServiceManager): Profile for oxz_switch: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/mgs"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12225'/> <propval type="astring" name="id" value='b4c6208c-1aaa-4d66-bcea-5b459f1a3b81'/> <propval type="astring" name="rack_id" value='6101b963-ee1b-430c-ad60-5a5922d708b7'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/wicketd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12226'/> <propval type="astring" name="artifact-address" value='[fdb0:18c0:4d0d:a02a::2]:12227'/> <propval type="astring" name="baseboard-file" value='/opt/oxide/baseboard.json'/> <propval type="astring" name="mgs-address" value='[::1]:12225'/> <propval type="astring" name="nexus-proxy-address" value='[::]:12229'/> <propval type="astring" name="rack-subnet" value='fd00:1122:3344::'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/switch_zone_setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="gz_local_link_addr" value='fe80::8:20ff:fe4c:7c81'/> <property type="astring" name="link_local_links"> <astring_list> <value_node value="tfportrear0_0" /> <value_node value="oxBootstrap0" /> </astring_list> </property> <propval type="astring" name="bootstrap_addr" value='fdb0:18c0:4d0d:a02a::2'/> <propval type="astring" name="bootstrap_vnic" value='oxBootstrap0'/> <propval type="astring" name="baseboard_info" value='{"type":"pc","identifier":"sock","model":"i86pc"}'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/dendrite"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_id" value='ea0e0fc4-fc93-4de6-b687-98567d824e83'/> <propval type="astring" name="rack_id" value='6101b963-ee1b-430c-ad60-5a5922d708b7'/> <propval type="astring" name="address" value='[::1]:12224'/> <propval type="astring" name="front_ports" value='1'/> <propval type="astring" name="rear_ports" value='1'/> <propval type="astring" name="port_config" value='/opt/oxide/dendrite/misc/softnpu_single_sled_config.toml'/> <propval type="astring" name="mgmt" value='uds'/> <propval type="astring" name="uds_path" value='/opt/softnpu/stuff'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/tfport"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="host" value='[::1]'/> <propval type="astring" name="port" value='12224'/> <propval type="astring" name="flags" value='--sync-only'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/lldpd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="board_rev" value='softnpu_front_1_rear_1'/> <propval type="astring" name="scrimlet_id" value='sock'/> <propval type="astring" name="scrimlet_model" value='i86pc'/> <propval type="astring" name="address" value='[::1]:12230'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/pumpkind"> </service> <service version="1" type="service" name="oxide/mgd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_uuid" value='ea0e0fc4-fc93-4de6-b687-98567d824e83'/> <propval type="astring" name="rack_uuid" value='6101b963-ee1b-430c-ad60-5a5922d708b7'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/mg-ddm"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="mode" value='transit'/> <propval type="astring" name="dendrite" value='true'/> <propval type="astring" name="sled_uuid" value='ea0e0fc4-fc93-4de6-b687-98567d824e83'/> <propval type="astring" name="rack_uuid" value='6101b963-ee1b-430c-ad60-5a5922d708b7'/> <propval type="astring" name="interfaces" value='tfportrear0_0/ll'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/uplink"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
2612024-07-26T02:33:59.214ZINFOSledAgent (ServiceManager): Booting oxz_switch zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_switch
2622024-07-26T02:33:59.372ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8/crypt/zone file = sled-storage/src/dataset.rs:411
2632024-07-26T02:33:59.978ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 70.800944ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2642024-07-26T02:34:00.384ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/ea0e0fc4-fc93-4de6-b687-98567d824e83): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
2652024-07-26T02:34:00.608ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 74.974251ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2662024-07-26T02:34:01.347ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 235.267469ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2672024-07-26T02:34:01.350ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/ea0e0fc4-fc93-4de6-b687-98567d824e83): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:600 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
2682024-07-26T02:34:01.998ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 220.677335ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2692024-07-26T02:34:02.460ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 4.403578651s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
2702024-07-26T02:34:02.460ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/ea0e0fc4-fc93-4de6-b687-98567d824e83): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
2712024-07-26T02:34:02.461ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 737.195215ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2722024-07-26T02:34:02.704ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 file = sled-storage/src/dataset.rs:496
2732024-07-26T02:34:02.920ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:512
2742024-07-26T02:34:02.920ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:429
2752024-07-26T02:34:02.920ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:414
2762024-07-26T02:34:02.920ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
2772024-07-26T02:34:02.920ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } path = /scratch/u2_1.vdev
2782024-07-26T02:34:03.155ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_1.vdev file = sled-hardware/src/disk.rs:360
2792024-07-26T02:34:03.156ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool)
2802024-07-26T02:34:04.354ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:304 zpool = ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }
2812024-07-26T02:34:04.608ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:348
2822024-07-26T02:34:04.608ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:350
2832024-07-26T02:34:04.608ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:354
2842024-07-26T02:34:04.608ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:356
2852024-07-26T02:34:04.608ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2862024-07-26T02:34:04.608ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2872024-07-26T02:34:04.608ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:368
2882024-07-26T02:34:05.145ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/ea0e0fc4-fc93-4de6-b687-98567d824e83): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:600 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
2892024-07-26T02:34:05.145ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/ea0e0fc4-fc93-4de6-b687-98567d824e83): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
2902024-07-26T02:34:05.851ZINFOSledAgent (ServiceManager): Re-enabling running switch zone (new address) file = sled-agent/src/services.rs:3919 new = [fd00:1122:3344:101::2, ::1] old = [::1]
2912024-07-26T02:34:05.851ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists file = sled-agent/src/services.rs:3935
2922024-07-26T02:34:05.851ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::2, prefix: 64 })) file = illumos-utils/src/running_zone.rs:518 zone = oxz_switch
2932024-07-26T02:34:06.075ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2942024-07-26T02:34:06.304ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/ea0e0fc4-fc93-4de6-b687-98567d824e83): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
2952024-07-26T02:34:06.310ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_b8ef9471-5b81-40ed-90b5-1103a4a9eb91/crypt/zone file = sled-storage/src/dataset.rs:411
2962024-07-26T02:34:06.788ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/ea0e0fc4-fc93-4de6-b687-98567d824e83): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:600 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
2972024-07-26T02:34:07.030ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 7.220367583s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
2982024-07-26T02:34:07.486ZINFOSledAgent (VmmReservoirManager): Finished setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:279 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
2992024-07-26T02:34:07.488ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/ea0e0fc4-fc93-4de6-b687-98567d824e83): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
3002024-07-26T02:34:07.569ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_b8ef9471-5b81-40ed-90b5-1103a4a9eb91 file = sled-storage/src/dataset.rs:496
3012024-07-26T02:34:07.576ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:512
3022024-07-26T02:34:07.576ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:429
3032024-07-26T02:34:07.576ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:414
3042024-07-26T02:34:07.576ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
3052024-07-26T02:34:07.576ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } path = /scratch/u2_2.vdev
3062024-07-26T02:34:07.585ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_2.vdev file = sled-hardware/src/disk.rs:360
3072024-07-26T02:34:07.585ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = a47af5d0-e565-4513-84ad-c553b7265a72 (zpool)
3082024-07-26T02:34:07.682ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:304 zpool = ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }
3092024-07-26T02:34:07.690ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:348
3102024-07-26T02:34:07.690ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:350
3112024-07-26T02:34:07.690ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:354
3122024-07-26T02:34:07.690ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:356
3132024-07-26T02:34:07.690ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3142024-07-26T02:34:07.690ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3152024-07-26T02:34:07.690ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:368
3162024-07-26T02:34:07.726ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3172024-07-26T02:34:07.734ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_a47af5d0-e565-4513-84ad-c553b7265a72/crypt/zone file = sled-storage/src/dataset.rs:411
3182024-07-26T02:34:07.858ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_a47af5d0-e565-4513-84ad-c553b7265a72 file = sled-storage/src/dataset.rs:496
3192024-07-26T02:34:07.867ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:512
3202024-07-26T02:34:07.867ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:429
3212024-07-26T02:34:07.867ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:414
3222024-07-26T02:34:07.867ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
3232024-07-26T02:34:07.867ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } path = /scratch/u2_3.vdev
3242024-07-26T02:34:07.877ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_3.vdev file = sled-hardware/src/disk.rs:360
3252024-07-26T02:34:07.877ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool)
3262024-07-26T02:34:07.975ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:304 zpool = ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }
3272024-07-26T02:34:07.983ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:348
3282024-07-26T02:34:07.983ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:350
3292024-07-26T02:34:07.983ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:354
3302024-07-26T02:34:07.983ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:356
3312024-07-26T02:34:07.983ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3322024-07-26T02:34:07.983ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3332024-07-26T02:34:07.983ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:368
3342024-07-26T02:34:08.018ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3352024-07-26T02:34:08.027ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_2a33f7b1-2bfd-4239-9138-fb534f4ccb3f/crypt/zone file = sled-storage/src/dataset.rs:411
3362024-07-26T02:34:08.151ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_2a33f7b1-2bfd-4239-9138-fb534f4ccb3f file = sled-storage/src/dataset.rs:496
3372024-07-26T02:34:08.160ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:512
3382024-07-26T02:34:08.160ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:429
3392024-07-26T02:34:08.160ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:414
3402024-07-26T02:34:08.160ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
3412024-07-26T02:34:08.160ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } path = /scratch/u2_4.vdev
3422024-07-26T02:34:08.169ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_4.vdev file = sled-hardware/src/disk.rs:360
3432024-07-26T02:34:08.169ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool)
3442024-07-26T02:34:08.264ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:304 zpool = ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }
3452024-07-26T02:34:08.272ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:348
3462024-07-26T02:34:08.272ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:350
3472024-07-26T02:34:08.272ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:354
3482024-07-26T02:34:08.272ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:356
3492024-07-26T02:34:08.272ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3502024-07-26T02:34:08.272ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3512024-07-26T02:34:08.272ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:368
3522024-07-26T02:34:08.307ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3532024-07-26T02:34:08.315ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_a38a1ea7-44a1-411f-bbb8-767c9a23135f/crypt/zone file = sled-storage/src/dataset.rs:411
3542024-07-26T02:34:08.441ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_a38a1ea7-44a1-411f-bbb8-767c9a23135f file = sled-storage/src/dataset.rs:496
3552024-07-26T02:34:08.449ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:512
3562024-07-26T02:34:08.449ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:429
3572024-07-26T02:34:08.449ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:414
3582024-07-26T02:34:08.449ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
3592024-07-26T02:34:08.449ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } path = /scratch/u2_5.vdev
3602024-07-26T02:34:08.458ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_5.vdev file = sled-hardware/src/disk.rs:360
3612024-07-26T02:34:08.458ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = cb102b97-0352-4c03-8d37-4c80c4475510 (zpool)
3622024-07-26T02:34:08.554ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:304 zpool = ZpoolName { id: cb102b97-0352-4c03-8d37-4c80c4475510 (zpool), kind: External }
3632024-07-26T02:34:08.562ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:348
3642024-07-26T02:34:08.562ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:350
3652024-07-26T02:34:08.562ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:354
3662024-07-26T02:34:08.562ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:356
3672024-07-26T02:34:08.562ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3682024-07-26T02:34:08.562ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3692024-07-26T02:34:08.562ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:368
3702024-07-26T02:34:08.566ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/ea0e0fc4-fc93-4de6-b687-98567d824e83): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
3712024-07-26T02:34:08.600ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3722024-07-26T02:34:08.607ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_cb102b97-0352-4c03-8d37-4c80c4475510/crypt/zone file = sled-storage/src/dataset.rs:411
3732024-07-26T02:34:08.612ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/ea0e0fc4-fc93-4de6-b687-98567d824e83): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:600 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
3742024-07-26T02:34:08.736ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_cb102b97-0352-4c03-8d37-4c80c4475510 file = sled-storage/src/dataset.rs:496
3752024-07-26T02:34:08.744ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:512
3762024-07-26T02:34:08.744ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:429
3772024-07-26T02:34:08.744ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:414
3782024-07-26T02:34:08.744ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
3792024-07-26T02:34:08.744ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } path = /scratch/u2_6.vdev
3802024-07-26T02:34:08.754ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_6.vdev file = sled-hardware/src/disk.rs:360
3812024-07-26T02:34:08.754ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = c5566b16-9a9f-498b-a4b1-83ac43308db5 (zpool)
3822024-07-26T02:34:08.853ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:304 zpool = ZpoolName { id: c5566b16-9a9f-498b-a4b1-83ac43308db5 (zpool), kind: External }
3832024-07-26T02:34:08.861ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:348
3842024-07-26T02:34:08.861ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:350
3852024-07-26T02:34:08.861ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:354
3862024-07-26T02:34:08.861ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:356
3872024-07-26T02:34:08.862ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3882024-07-26T02:34:08.862ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3892024-07-26T02:34:08.862ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:368
3902024-07-26T02:34:08.897ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3912024-07-26T02:34:08.905ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_c5566b16-9a9f-498b-a4b1-83ac43308db5/crypt/zone file = sled-storage/src/dataset.rs:411
3922024-07-26T02:34:09.027ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_c5566b16-9a9f-498b-a4b1-83ac43308db5 file = sled-storage/src/dataset.rs:496
3932024-07-26T02:34:09.035ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:512
3942024-07-26T02:34:09.035ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:429
3952024-07-26T02:34:09.035ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:414
3962024-07-26T02:34:09.035ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
3972024-07-26T02:34:09.035ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } path = /scratch/u2_7.vdev
3982024-07-26T02:34:09.045ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_7.vdev file = sled-hardware/src/disk.rs:360
3992024-07-26T02:34:09.045ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 136ed548-ab63-44da-a187-0fa13dc111a0 (zpool)
4002024-07-26T02:34:09.144ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:304 zpool = ZpoolName { id: 136ed548-ab63-44da-a187-0fa13dc111a0 (zpool), kind: External }
4012024-07-26T02:34:09.154ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:348
4022024-07-26T02:34:09.154ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:350
4032024-07-26T02:34:09.154ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:354
4042024-07-26T02:34:09.154ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:356
4052024-07-26T02:34:09.154ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4062024-07-26T02:34:09.154ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4072024-07-26T02:34:09.154ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:368
4082024-07-26T02:34:09.195ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4092024-07-26T02:34:09.203ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_136ed548-ab63-44da-a187-0fa13dc111a0/crypt/zone file = sled-storage/src/dataset.rs:411
4102024-07-26T02:34:09.329ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_136ed548-ab63-44da-a187-0fa13dc111a0 file = sled-storage/src/dataset.rs:496
4112024-07-26T02:34:09.336ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:512
4122024-07-26T02:34:09.336ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:429
4132024-07-26T02:34:09.336ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:414
4142024-07-26T02:34:09.336ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
4152024-07-26T02:34:09.336ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } path = /scratch/u2_8.vdev
4162024-07-26T02:34:09.346ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_8.vdev file = sled-hardware/src/disk.rs:360
4172024-07-26T02:34:09.346ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 3ed9ecd9-4aeb-4ec2-8876-b9c8ed7df016 (zpool)
4182024-07-26T02:34:09.442ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:304 zpool = ZpoolName { id: 3ed9ecd9-4aeb-4ec2-8876-b9c8ed7df016 (zpool), kind: External }
4192024-07-26T02:34:09.450ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:348
4202024-07-26T02:34:09.450ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:350
4212024-07-26T02:34:09.450ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:354
4222024-07-26T02:34:09.450ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:356
4232024-07-26T02:34:09.450ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4242024-07-26T02:34:09.450ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4252024-07-26T02:34:09.450ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:368
4262024-07-26T02:34:09.487ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4272024-07-26T02:34:09.496ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_3ed9ecd9-4aeb-4ec2-8876-b9c8ed7df016/crypt/zone file = sled-storage/src/dataset.rs:411
4282024-07-26T02:34:09.569ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/ea0e0fc4-fc93-4de6-b687-98567d824e83): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
4292024-07-26T02:34:09.627ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_3ed9ecd9-4aeb-4ec2-8876-b9c8ed7df016 file = sled-storage/src/dataset.rs:496
4302024-07-26T02:34:09.635ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:512
4312024-07-26T02:34:09.635ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:429
4322024-07-26T02:34:09.635ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4332024-07-26T02:34:09.635ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f09c5312-7869-4743-8a0d-863eb3bf0156 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: cb102b97-0352-4c03-8d37-4c80c4475510 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: c5566b16-9a9f-498b-a4b1-83ac43308db5 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 136ed548-ab63-44da-a187-0fa13dc111a0 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 3ed9ecd9-4aeb-4ec2-8876-b9c8ed7df016 (zpool), kind: External } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
4342024-07-26T02:34:09.635ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
4352024-07-26T02:34:09.635ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
4362024-07-26T02:34:09.635ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4372024-07-26T02:34:09.636ZINFOSledAgent: physical disks ensure: Updated storage file = sled-agent/src/sled_agent.rs:831 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
4382024-07-26T02:34:09.636ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
4392024-07-26T02:34:09.636ZINFOSledAgent: physical disks ensure: Propagating new generation of disks file = sled-agent/src/sled_agent.rs:857 generation = Generation(1) sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
4402024-07-26T02:34:09.636ZINFOSledAgent: physical disks ensure: Updated storage monitor file = sled-agent/src/sled_agent.rs:862 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
4412024-07-26T02:34:09.636ZINFOSledAgent: physical disks ensure: Updated zone bundler file = sled-agent/src/sled_agent.rs:867 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
4422024-07-26T02:34:09.636ZINFOSledAgent (ProbeManager): use_only_these_disks: Processing new request file = sled-agent/src/probe_manager.rs:116 gen = Generation(1) sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
4432024-07-26T02:34:09.636ZINFOSledAgent: physical disks ensure: Updated probes file = sled-agent/src/sled_agent.rs:873 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
4442024-07-26T02:34:09.636ZINFOSledAgent (InstanceManager): use_only_these_disks: Processing new request file = sled-agent/src/instance_manager.rs:822 gen = Generation(1)
4452024-07-26T02:34:09.636ZINFOSledAgent: physical disks ensure: Updated instances file = sled-agent/src/sled_agent.rs:878 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
4462024-07-26T02:34:09.636ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 12395250 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:44735 req_id = 665f7da0-e4dc-41e9-9c56-88ff6ac7ee13 response_code = 200 uri = /omicron-physical-disks
4472024-07-26T02:34:09.666ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(2), zones: [OmicronZoneConfig { id: 25724e7e-f0f3-413d-ab41-b4a33973050b, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 82379658-84e2-487b-bcd4-b30ffc1d9874, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
4482024-07-26T02:34:09.666ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:52523
4492024-07-26T02:34:09.667ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
4502024-07-26T02:34:09.667ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4512024-07-26T02:34:09.720ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
4522024-07-26T02:34:09.720ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4532024-07-26T02:34:09.769ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
4542024-07-26T02:34:09.769ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4552024-07-26T02:34:09.819ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
4562024-07-26T02:34:09.819ZINFOSledAgent (ServiceManager): No ledger in /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:188
4572024-07-26T02:34:09.819ZINFOSledAgent (ServiceManager): No ledger in /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:188
4582024-07-26T02:34:09.819ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
4592024-07-26T02:34:09.863ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
4602024-07-26T02:34:09.863ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
4612024-07-26T02:34:09.863ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
4622024-07-26T02:34:09.914ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_4e64b0cb-e3e2-4959-adf4-d5cd70f25b40 file = illumos-utils/src/zone.rs:316
4632024-07-26T02:34:09.917ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_82379658-84e2-487b-bcd4-b30ffc1d9874 file = illumos-utils/src/zone.rs:316
4642024-07-26T02:34:09.921ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_25724e7e-f0f3-413d-ab41-b4a33973050b file = illumos-utils/src/zone.rs:316
4652024-07-26T02:34:09.933ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_4e64b0cb-e3e2-4959-adf4-d5cd70f25b40 file = illumos-utils/src/zone.rs:354
4662024-07-26T02:34:09.938ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_82379658-84e2-487b-bcd4-b30ffc1d9874 file = illumos-utils/src/zone.rs:354
4672024-07-26T02:34:09.941ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_25724e7e-f0f3-413d-ab41-b4a33973050b file = illumos-utils/src/zone.rs:354
4682024-07-26T02:34:10.572ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/ea0e0fc4-fc93-4de6-b687-98567d824e83): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
4692024-07-26T02:34:10.611ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/ea0e0fc4-fc93-4de6-b687-98567d824e83): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:600 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
4702024-07-26T02:34:11.403ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_82379658-84e2-487b-bcd4-b30ffc1d9874: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService2'/> <propval type="astring" name="gateway" value='fd00:1122:3344:3::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:3::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:3::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:3::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4712024-07-26T02:34:11.403ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:3::, width: 64 }
4722024-07-26T02:34:11.883ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_4e64b0cb-e3e2-4959-adf4-d5cd70f25b40: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService1'/> <propval type="astring" name="gateway" value='fd00:1122:3344:2::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:2::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:2::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:2::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4732024-07-26T02:34:11.883ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_82379658-84e2-487b-bcd4-b30ffc1d9874 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_internal_dns_82379658-84e2-487b-bcd4-b30ffc1d9874
4742024-07-26T02:34:11.887ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:2::, width: 64 }
4752024-07-26T02:34:12.367ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_25724e7e-f0f3-413d-ab41-b4a33973050b: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService3'/> <propval type="astring" name="gateway" value='fd00:1122:3344:1::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:1::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:1::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:1::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4762024-07-26T02:34:12.367ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_4e64b0cb-e3e2-4959-adf4-d5cd70f25b40 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_internal_dns_4e64b0cb-e3e2-4959-adf4-d5cd70f25b40
4772024-07-26T02:34:12.381ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_82379658-84e2-487b-bcd4-b30ffc1d9874") failed: Property not found. retry in 54.895447ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_82379658-84e2-487b-bcd4-b30ffc1d9874
4782024-07-26T02:34:12.381ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:1::, width: 64 }
4792024-07-26T02:34:12.381ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_25724e7e-f0f3-413d-ab41-b4a33973050b zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_internal_dns_25724e7e-f0f3-413d-ab41-b4a33973050b
4802024-07-26T02:34:12.444ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_82379658-84e2-487b-bcd4-b30ffc1d9874") failed: Property not found. retry in 92.735536ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_82379658-84e2-487b-bcd4-b30ffc1d9874
4812024-07-26T02:34:12.492ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_4e64b0cb-e3e2-4959-adf4-d5cd70f25b40") failed: Property not found. retry in 35.009631ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_4e64b0cb-e3e2-4959-adf4-d5cd70f25b40
4822024-07-26T02:34:12.509ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_25724e7e-f0f3-413d-ab41-b4a33973050b") failed: Property not found. retry in 29.078247ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_25724e7e-f0f3-413d-ab41-b4a33973050b
4832024-07-26T02:34:12.535ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_4e64b0cb-e3e2-4959-adf4-d5cd70f25b40") failed: Property not found. retry in 137.551517ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_4e64b0cb-e3e2-4959-adf4-d5cd70f25b40
4842024-07-26T02:34:12.545ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_82379658-84e2-487b-bcd4-b30ffc1d9874") failed: Property not found. retry in 166.654458ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_82379658-84e2-487b-bcd4-b30ffc1d9874
4852024-07-26T02:34:12.551ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_25724e7e-f0f3-413d-ab41-b4a33973050b") failed: Property not found. retry in 148.233182ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_25724e7e-f0f3-413d-ab41-b4a33973050b
4862024-07-26T02:34:12.716ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_4e64b0cb-e3e2-4959-adf4-d5cd70f25b40") failed: Property not found. retry in 288.670036ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_4e64b0cb-e3e2-4959-adf4-d5cd70f25b40
4872024-07-26T02:34:12.780ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_25724e7e-f0f3-413d-ab41-b4a33973050b") failed: Property not found. retry in 284.010617ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_25724e7e-f0f3-413d-ab41-b4a33973050b
4882024-07-26T02:34:12.812ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_82379658-84e2-487b-bcd4-b30ffc1d9874") failed: Property not found. retry in 410.349169ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_82379658-84e2-487b-bcd4-b30ffc1d9874
4892024-07-26T02:34:13.018ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_4e64b0cb-e3e2-4959-adf4-d5cd70f25b40") failed: Property not found. retry in 526.575754ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_4e64b0cb-e3e2-4959-adf4-d5cd70f25b40
4902024-07-26T02:34:13.077ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_25724e7e-f0f3-413d-ab41-b4a33973050b") failed: Property not found. retry in 400.636883ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_25724e7e-f0f3-413d-ab41-b4a33973050b
4912024-07-26T02:34:13.232ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_82379658-84e2-487b-bcd4-b30ffc1d9874") failed: Property not found. retry in 707.414186ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_82379658-84e2-487b-bcd4-b30ffc1d9874
4922024-07-26T02:34:13.486ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_25724e7e-f0f3-413d-ab41-b4a33973050b") failed: Property not found. retry in 635.643165ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_25724e7e-f0f3-413d-ab41-b4a33973050b
4932024-07-26T02:34:13.554ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_4e64b0cb-e3e2-4959-adf4-d5cd70f25b40") failed: Property not found. retry in 1.078079327s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_4e64b0cb-e3e2-4959-adf4-d5cd70f25b40
4942024-07-26T02:34:13.950ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_82379658-84e2-487b-bcd4-b30ffc1d9874") failed: Property not found. retry in 833.988866ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_82379658-84e2-487b-bcd4-b30ffc1d9874
4952024-07-26T02:34:14.134ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_25724e7e-f0f3-413d-ab41-b4a33973050b") failed: Property not found. retry in 1.363184546s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_25724e7e-f0f3-413d-ab41-b4a33973050b
4962024-07-26T02:34:14.641ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_4e64b0cb-e3e2-4959-adf4-d5cd70f25b40") failed: Property not found. retry in 702.161213ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_4e64b0cb-e3e2-4959-adf4-d5cd70f25b40
4972024-07-26T02:34:14.793ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_82379658-84e2-487b-bcd4-b30ffc1d9874") failed: Property not found. retry in 1.255473366s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_82379658-84e2-487b-bcd4-b30ffc1d9874
4982024-07-26T02:34:15.350ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_4e64b0cb-e3e2-4959-adf4-d5cd70f25b40") failed: Property not found. retry in 1.33036558s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_4e64b0cb-e3e2-4959-adf4-d5cd70f25b40
4992024-07-26T02:34:15.505ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_25724e7e-f0f3-413d-ab41-b4a33973050b") failed: Property not found. retry in 813.825333ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_25724e7e-f0f3-413d-ab41-b4a33973050b
5002024-07-26T02:34:15.675ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists - OK file = sled-agent/src/services.rs:3943
5012024-07-26T02:34:15.675ZINFOSledAgent (ServiceManager): Ensuring there is a default route file = sled-agent/src/services.rs:3953 gateway = fd00:1122:3344:101::1
5022024-07-26T02:34:15.680ZINFOSledAgent (ServiceManager): configuring dendrite service file = sled-agent/src/services.rs:4032
5032024-07-26T02:34:15.805ZINFOSledAgent (ServiceManager): refreshed dendrite service with new configuration file = sled-agent/src/services.rs:4080
5042024-07-26T02:34:15.805ZINFOSledAgent (ServiceManager): configuring lldp service file = sled-agent/src/services.rs:4107
5052024-07-26T02:34:15.844ZINFOSledAgent (ServiceManager): refreshed lldpd service with new configuration file = sled-agent/src/services.rs:4120
5062024-07-26T02:34:15.844ZINFOSledAgent (ServiceManager): configuring MGS service file = sled-agent/src/services.rs:3984
5072024-07-26T02:34:15.897ZINFOSledAgent (ServiceManager): refreshed MGS service with new configuration file = sled-agent/src/services.rs:4026
5082024-07-26T02:34:15.897ZINFOSledAgent (ServiceManager): configuring wicketd file = sled-agent/src/services.rs:4087 rack_subnet = fd00:1122:3344::
5092024-07-26T02:34:15.917ZINFOSledAgent (ServiceManager): refreshed wicketd service with new configuration file = sled-agent/src/services.rs:4098
5102024-07-26T02:34:15.917ZINFOSledAgent (ServiceManager): configuring mgd service file = sled-agent/src/services.rs:4140
5112024-07-26T02:34:16.004ZINFOSledAgent (ServiceManager): refreshed mgd service with new configuration file = sled-agent/src/services.rs:4172
5122024-07-26T02:34:16.004ZINFOSledAgent (ServiceManager): configuring mg-ddm service file = sled-agent/src/services.rs:4178
5132024-07-26T02:34:16.057ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_82379658-84e2-487b-bcd4-b30ffc1d9874") failed: Property not found. retry in 1.097771799s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_82379658-84e2-487b-bcd4-b30ffc1d9874
5142024-07-26T02:34:16.119ZINFOSledAgent (ServiceManager): refreshed mg-ddm service with new configuration file = sled-agent/src/services.rs:4219
5152024-07-26T02:34:16.119ZINFOSledAgent (ServiceManager): Determining physical location of our switch zone at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:364
5162024-07-26T02:34:16.151ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 55.901202ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/bootstrap/early_networking.rs:383
5172024-07-26T02:34:16.208ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 64.744537ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/bootstrap/early_networking.rs:383
5182024-07-26T02:34:16.273ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 239.518105ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/bootstrap/early_networking.rs:383
5192024-07-26T02:34:16.328ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_25724e7e-f0f3-413d-ab41-b4a33973050b") failed: Property not found. retry in 561.947656ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_25724e7e-f0f3-413d-ab41-b4a33973050b
5202024-07-26T02:34:16.515ZINFOSledAgent (ServiceManager): Initializing 1 Uplinks on Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:412
5212024-07-26T02:34:16.546ZINFOSledAgent (ServiceManager): Building Port Configuration file = sled-agent/src/bootstrap/early_networking.rs:651
5222024-07-26T02:34:16.547ZINFOSledAgent (ServiceManager): Checking dendrite uptime file = sled-agent/src/bootstrap/early_networking.rs:689
5232024-07-26T02:34:16.547ZINFOSledAgent (ServiceManager): Dendrite online file = sled-agent/src/bootstrap/early_networking.rs:692 uptime = 0
5242024-07-26T02:34:16.547ZINFOSledAgent (ServiceManager): Configuring default uplink on switch config = PortSettings {\n links: {\n "0": LinkSettings {\n addrs: [\n 10.151.2.174,\n ],\n params: LinkCreate {\n autoneg: false,\n fec: None,\n kr: false,\n lane: Some(\n LinkId(\n 0,\n ),\n ),\n speed: Speed40G,\n },\n },\n },\n} file = sled-agent/src/bootstrap/early_networking.rs:434
5252024-07-26T02:34:16.585ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
5262024-07-26T02:34:16.606ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
5272024-07-26T02:34:16.625ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:405
5282024-07-26T02:34:16.625ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5292024-07-26T02:34:16.625ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5302024-07-26T02:34:16.625ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5312024-07-26T02:34:16.625ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5322024-07-26T02:34:16.625ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5332024-07-26T02:34:16.625ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5342024-07-26T02:34:16.625ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5352024-07-26T02:34:16.625ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5362024-07-26T02:34:16.625ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5372024-07-26T02:34:16.625ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5382024-07-26T02:34:16.625ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5392024-07-26T02:34:16.625ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5402024-07-26T02:34:16.625ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5412024-07-26T02:34:16.625ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5422024-07-26T02:34:16.625ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5432024-07-26T02:34:16.625ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5442024-07-26T02:34:16.625ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5452024-07-26T02:34:16.625ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5462024-07-26T02:34:16.625ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5472024-07-26T02:34:16.625ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5482024-07-26T02:34:16.625ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5492024-07-26T02:34:16.625ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5502024-07-26T02:34:16.690ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_4e64b0cb-e3e2-4959-adf4-d5cd70f25b40") failed: Property not found. retry in 914.115801ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_4e64b0cb-e3e2-4959-adf4-d5cd70f25b40
5512024-07-26T02:34:16.898ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_25724e7e-f0f3-413d-ab41-b4a33973050b") failed: Property not found. retry in 1.201890722s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_25724e7e-f0f3-413d-ab41-b4a33973050b
5522024-07-26T02:34:17.166ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_82379658-84e2-487b-bcd4-b30ffc1d9874") failed: Property not found. retry in 734.672578ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_82379658-84e2-487b-bcd4-b30ffc1d9874
5532024-07-26T02:34:18.118ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3058 zone = oxz_internal_dns_25724e7e-f0f3-413d-ab41-b4a33973050b
5542024-07-26T02:34:18.119ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3058 zone = oxz_internal_dns_4e64b0cb-e3e2-4959-adf4-d5cd70f25b40
5552024-07-26T02:34:18.119ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3058 zone = oxz_internal_dns_82379658-84e2-487b-bcd4-b30ffc1d9874
5562024-07-26T02:34:18.119ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5572024-07-26T02:34:18.119ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5582024-07-26T02:34:18.119ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 8452713 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52523 req_id = 57753561-9544-4d23-ab2b-bc0defcbb7ff response_code = 204 uri = /omicron-zones
5592024-07-26T02:34:18.121ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:1::1]:5353 file = sled-agent/src/rack_setup/service.rs:555
5602024-07-26T02:34:18.166ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:2::1]:5353 file = sled-agent/src/rack_setup/service.rs:555
5612024-07-26T02:34:18.213ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:3::1]:5353 file = sled-agent/src/rack_setup/service.rs:555
5622024-07-26T02:34:18.260ZINFOSledAgent (RSS): Configured all DNS servers file = sled-agent/src/rack_setup/service.rs:591
5632024-07-26T02:34:18.260ZINFOSledAgent (RSS): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:258
5642024-07-26T02:34:18.290ZINFOSledAgent (RSS): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:275
5652024-07-26T02:34:18.292ZINFOSledAgent (RSS): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:300
5662024-07-26T02:34:18.292ZINFOSledAgent (RSS): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:192
5672024-07-26T02:34:18.322ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(3), zones: [OmicronZoneConfig { id: 25724e7e-f0f3-413d-ab41-b4a33973050b, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 82379658-84e2-487b-bcd4-b30ffc1d9874, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0eb4388e-1e4e-49da-b41b-58746266c965, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bbb96e05-c4c2-4d74-93ab-32168e25b9fd, kind: Service { id: 0eb4388e-1e4e-49da-b41b-58746266c965 }, name: Name("ntp-0eb4388e-1e4e-49da-b41b-58746266c965"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 87])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
5682024-07-26T02:34:18.322ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:57497
5692024-07-26T02:34:18.322ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
5702024-07-26T02:34:18.322ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5712024-07-26T02:34:18.340ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
5722024-07-26T02:34:18.340ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5732024-07-26T02:34:18.355ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
5742024-07-26T02:34:18.355ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5752024-07-26T02:34:18.370ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
5762024-07-26T02:34:18.371ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
5772024-07-26T02:34:18.371ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
5782024-07-26T02:34:18.371ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
5792024-07-26T02:34:18.382ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:258
5802024-07-26T02:34:18.414ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:275
5812024-07-26T02:34:18.415ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:300
5822024-07-26T02:34:18.415ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:192
5832024-07-26T02:34:18.444ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:426 port = Port { inner: PortInner(PortData { name: "opte0", ip: 172.30.3.5, mac: MacAddr6([168, 64, 37, 255, 143, 87]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.3.1 } }) }
5842024-07-26T02:34:18.444ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1207 zone_type = boundary_ntp
5852024-07-26T02:34:18.445ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.169 file = clients/dpd-client/src/lib.rs:83
5862024-07-26T02:34:18.445ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.169 file = clients/dpd-client/src/lib.rs:90
5872024-07-26T02:34:18.445ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.169 file = clients/dpd-client/src/lib.rs:118
5882024-07-26T02:34:18.445ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
5892024-07-26T02:34:18.470ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_ntp_0eb4388e-1e4e-49da-b41b-58746266c965 file = illumos-utils/src/zone.rs:316
5902024-07-26T02:34:18.491ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_ntp_0eb4388e-1e4e-49da-b41b-58746266c965 file = illumos-utils/src/zone.rs:354
5912024-07-26T02:34:19.130ZINFOSledAgent (ServiceManager): Profile for oxz_ntp_0eb4388e-1e4e-49da-b41b-58746266c965: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService4'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/chrony-setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="allow" value='fd00:1122:3344:100::/56'/> <propval type="boolean" name="boundary" value='true'/> <propval type="astring" name="server" value='0.pool.ntp.org'/> </property_group> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="1.1.1.1" /> <value_node value="9.9.9.9" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/ntp"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte0'/> <propval type="astring" name="gateway" value='172.30.3.1'/> <propval type="astring" name="ip" value='172.30.3.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
5922024-07-26T02:34:19.130ZINFOSledAgent (ServiceManager): Booting oxz_ntp_0eb4388e-1e4e-49da-b41b-58746266c965 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_ntp_0eb4388e-1e4e-49da-b41b-58746266c965
5932024-07-26T02:34:19.296ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_0eb4388e-1e4e-49da-b41b-58746266c965") failed: Property not found. retry in 47.987927ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_0eb4388e-1e4e-49da-b41b-58746266c965
5942024-07-26T02:34:19.358ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_0eb4388e-1e4e-49da-b41b-58746266c965") failed: Property not found. retry in 115.677211ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_0eb4388e-1e4e-49da-b41b-58746266c965
5952024-07-26T02:34:19.499ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_0eb4388e-1e4e-49da-b41b-58746266c965") failed: Property not found. retry in 218.068044ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_0eb4388e-1e4e-49da-b41b-58746266c965
5962024-07-26T02:34:19.727ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_0eb4388e-1e4e-49da-b41b-58746266c965") failed: Property not found. retry in 354.678647ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_0eb4388e-1e4e-49da-b41b-58746266c965
5972024-07-26T02:34:20.094ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_0eb4388e-1e4e-49da-b41b-58746266c965") failed: Property not found. retry in 430.92825ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_0eb4388e-1e4e-49da-b41b-58746266c965
5982024-07-26T02:34:20.533ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_0eb4388e-1e4e-49da-b41b-58746266c965") failed: Property not found. retry in 1.313211245s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_0eb4388e-1e4e-49da-b41b-58746266c965
5992024-07-26T02:34:21.856ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_0eb4388e-1e4e-49da-b41b-58746266c965") failed: Property not found. retry in 1.310893582s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_0eb4388e-1e4e-49da-b41b-58746266c965
6002024-07-26T02:34:23.177ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_0eb4388e-1e4e-49da-b41b-58746266c965") failed: Property not found. retry in 931.291802ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_0eb4388e-1e4e-49da-b41b-58746266c965
6012024-07-26T02:34:24.131ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3058 zone = oxz_ntp_0eb4388e-1e4e-49da-b41b-58746266c965
6022024-07-26T02:34:24.131ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6032024-07-26T02:34:24.131ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6042024-07-26T02:34:24.131ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 5809479 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:57497 req_id = 215b99fa-1563-4eca-927b-cfb1230d8d7d response_code = 204 uri = /omicron-zones
6052024-07-26T02:34:24.133ZINFOSledAgent (RSS): Waiting for rack time synchronization file = sled-agent/src/rack_setup/service.rs:632
6062024-07-26T02:34:24.167ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6072024-07-26T02:34:24.167ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:59646
6082024-07-26T02:34:24.176ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_0eb4388e-1e4e-49da-b41b-58746266c965': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3647
6092024-07-26T02:34:24.176ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 9397 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:59646 req_id = 42651165-ac85-4d3f-a783-ab5a111aca6c response_code = 500 uri = /timesync
6102024-07-26T02:34:24.180ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:662
6112024-07-26T02:34:24.265ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6122024-07-26T02:34:24.265ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:44637
6132024-07-26T02:34:26.273ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_0eb4388e-1e4e-49da-b41b-58746266c965': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3647
6142024-07-26T02:34:26.273ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 2007550 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:44637 req_id = 6c452739-0f92-44c8-89cd-9794657f60db response_code = 500 uri = /timesync
6152024-07-26T02:34:26.276ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:662
6162024-07-26T02:34:26.441ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6172024-07-26T02:34:26.442ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:52868
6182024-07-26T02:34:26.452ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_0eb4388e-1e4e-49da-b41b-58746266c965': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3647
6192024-07-26T02:34:26.452ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 10168 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:52868 req_id = 9e57eb4e-92f4-4320-bd1f-d224a06dc569 response_code = 500 uri = /timesync
6202024-07-26T02:34:26.454ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:662
6212024-07-26T02:34:26.617ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6222024-07-26T02:34:26.617ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:38120
6232024-07-26T02:34:30.626ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_0eb4388e-1e4e-49da-b41b-58746266c965': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3647
6242024-07-26T02:34:30.626ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 4008918 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:38120 req_id = 41a23b36-8d29-43df-8453-53417e3e51f7 response_code = 500 uri = /timesync
6252024-07-26T02:34:30.629ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:662
6262024-07-26T02:34:30.816ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6272024-07-26T02:34:30.816ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:45930
6282024-07-26T02:34:30.825ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 8112 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:45930 req_id = 637425ba-a588-46a8-925e-1f098753b1f5 response_code = 200 uri = /timesync
6292024-07-26T02:34:30.828ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:640
6302024-07-26T02:34:30.828ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:662
6312024-07-26T02:34:31.055ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6322024-07-26T02:34:31.055ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:34251
6332024-07-26T02:34:31.076ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 20590 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:34251 req_id = 176c579e-3b3a-4e71-b3d9-e56002ae1dd2 response_code = 200 uri = /timesync
6342024-07-26T02:34:31.080ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: true, ref_id: 3330918944, ip_addr: 198.137.202.32, stratum: 3, ref_time: 1721961270.6833196, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:640
6352024-07-26T02:34:31.080ZINFOSledAgent (RSS): Finished setting up Internal DNS and NTP file = sled-agent/src/rack_setup/service.rs:1230
6362024-07-26T02:34:31.110ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 25724e7e-f0f3-413d-ab41-b4a33973050b, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 82379658-84e2-487b-bcd4-b30ffc1d9874, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0eb4388e-1e4e-49da-b41b-58746266c965, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bbb96e05-c4c2-4d74-93ab-32168e25b9fd, kind: Service { id: 0eb4388e-1e4e-49da-b41b-58746266c965 }, name: Name("ntp-0eb4388e-1e4e-49da-b41b-58746266c965"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 87])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } } } }, OmicronZoneConfig { id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
6372024-07-26T02:34:31.110ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:52946
6382024-07-26T02:34:31.111ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
6392024-07-26T02:34:31.111ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6402024-07-26T02:34:31.131ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
6412024-07-26T02:34:31.132ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6422024-07-26T02:34:31.150ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
6432024-07-26T02:34:31.150ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6442024-07-26T02:34:31.168ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
6452024-07-26T02:34:31.168ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6462024-07-26T02:34:31.221ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
6472024-07-26T02:34:31.221ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6482024-07-26T02:34:31.280ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
6492024-07-26T02:34:31.280ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6502024-07-26T02:34:32.013ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
6512024-07-26T02:34:32.013ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6522024-07-26T02:34:32.021ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
6532024-07-26T02:34:32.021ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6542024-07-26T02:34:32.073ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
6552024-07-26T02:34:32.073ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
6562024-07-26T02:34:32.074ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
6572024-07-26T02:34:32.074ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
6582024-07-26T02:34:32.106ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
6592024-07-26T02:34:32.106ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
6602024-07-26T02:34:32.106ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
6612024-07-26T02:34:32.106ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
6622024-07-26T02:34:32.106ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
6632024-07-26T02:34:32.106ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8", TimeNotSynchronized), ("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c", TimeNotSynchronized), ("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3", TimeNotSynchronized), ("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6", TimeNotSynchronized), ("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 361696 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52946 req_id = b7b8f59c-dce4-4936-9e3f-8c03b0b5572d response_code = 500 uri = /omicron-zones
6642024-07-26T02:34:32.106ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "b7b8f59c-dce4-4936-9e3f-8c03b0b5572d", "content-length": "124", "date": "Fri, 26 Jul 2024 02:34:32 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "b7b8f59c-dce4-4936-9e3f-8c03b0b5572d" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 57.957421ms sled_address = [fd00:1122:3344:101::1]:12345
6652024-07-26T02:34:32.166ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 25724e7e-f0f3-413d-ab41-b4a33973050b, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 82379658-84e2-487b-bcd4-b30ffc1d9874, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0eb4388e-1e4e-49da-b41b-58746266c965, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bbb96e05-c4c2-4d74-93ab-32168e25b9fd, kind: Service { id: 0eb4388e-1e4e-49da-b41b-58746266c965 }, name: Name("ntp-0eb4388e-1e4e-49da-b41b-58746266c965"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 87])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } } } }, OmicronZoneConfig { id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
6662024-07-26T02:34:32.166ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
6672024-07-26T02:34:32.166ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6682024-07-26T02:34:32.185ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
6692024-07-26T02:34:32.185ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6702024-07-26T02:34:32.202ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
6712024-07-26T02:34:32.202ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6722024-07-26T02:34:32.220ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
6732024-07-26T02:34:32.220ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6742024-07-26T02:34:32.237ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
6752024-07-26T02:34:32.237ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6762024-07-26T02:34:32.256ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
6772024-07-26T02:34:32.256ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6782024-07-26T02:34:32.274ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
6792024-07-26T02:34:32.274ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6802024-07-26T02:34:32.292ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
6812024-07-26T02:34:32.292ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6822024-07-26T02:34:32.310ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
6832024-07-26T02:34:32.310ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
6842024-07-26T02:34:32.310ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
6852024-07-26T02:34:32.310ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
6862024-07-26T02:34:32.341ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
6872024-07-26T02:34:32.341ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
6882024-07-26T02:34:32.341ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
6892024-07-26T02:34:32.341ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
6902024-07-26T02:34:32.341ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
6912024-07-26T02:34:32.341ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3", TimeNotSynchronized), ("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63", TimeNotSynchronized), ("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6", TimeNotSynchronized), ("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c", TimeNotSynchronized), ("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 174936 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52946 req_id = 45bf417b-cf26-481d-8388-dbba3ea42290 response_code = 500 uri = /omicron-zones
6922024-07-26T02:34:32.341ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "45bf417b-cf26-481d-8388-dbba3ea42290", "content-length": "124", "date": "Fri, 26 Jul 2024 02:34:32 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "45bf417b-cf26-481d-8388-dbba3ea42290" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 164.887512ms sled_address = [fd00:1122:3344:101::1]:12345
6932024-07-26T02:34:32.507ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 25724e7e-f0f3-413d-ab41-b4a33973050b, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 82379658-84e2-487b-bcd4-b30ffc1d9874, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0eb4388e-1e4e-49da-b41b-58746266c965, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bbb96e05-c4c2-4d74-93ab-32168e25b9fd, kind: Service { id: 0eb4388e-1e4e-49da-b41b-58746266c965 }, name: Name("ntp-0eb4388e-1e4e-49da-b41b-58746266c965"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 87])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } } } }, OmicronZoneConfig { id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
6942024-07-26T02:34:32.508ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
6952024-07-26T02:34:32.508ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6962024-07-26T02:34:32.527ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
6972024-07-26T02:34:32.527ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6982024-07-26T02:34:32.545ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
6992024-07-26T02:34:32.545ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7002024-07-26T02:34:32.561ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7012024-07-26T02:34:32.561ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7022024-07-26T02:34:32.579ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7032024-07-26T02:34:32.579ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7042024-07-26T02:34:32.596ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7052024-07-26T02:34:32.596ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7062024-07-26T02:34:32.614ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7072024-07-26T02:34:32.614ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7082024-07-26T02:34:32.632ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7092024-07-26T02:34:32.632ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7102024-07-26T02:34:32.649ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
7112024-07-26T02:34:32.649ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
7122024-07-26T02:34:32.649ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
7132024-07-26T02:34:32.650ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
7142024-07-26T02:34:32.680ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
7152024-07-26T02:34:32.680ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
7162024-07-26T02:34:32.680ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
7172024-07-26T02:34:32.680ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
7182024-07-26T02:34:32.680ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
7192024-07-26T02:34:32.680ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6", TimeNotSynchronized), ("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c", TimeNotSynchronized), ("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63", TimeNotSynchronized), ("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3", TimeNotSynchronized), ("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 172371 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52946 req_id = 7fde6d93-3c8f-4e89-9295-3bad561f0436 response_code = 500 uri = /omicron-zones
7202024-07-26T02:34:32.680ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "7fde6d93-3c8f-4e89-9295-3bad561f0436", "content-length": "124", "date": "Fri, 26 Jul 2024 02:34:32 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "7fde6d93-3c8f-4e89-9295-3bad561f0436" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 85.587557ms sled_address = [fd00:1122:3344:101::1]:12345
7212024-07-26T02:34:32.766ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 25724e7e-f0f3-413d-ab41-b4a33973050b, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 82379658-84e2-487b-bcd4-b30ffc1d9874, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0eb4388e-1e4e-49da-b41b-58746266c965, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bbb96e05-c4c2-4d74-93ab-32168e25b9fd, kind: Service { id: 0eb4388e-1e4e-49da-b41b-58746266c965 }, name: Name("ntp-0eb4388e-1e4e-49da-b41b-58746266c965"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 87])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } } } }, OmicronZoneConfig { id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
7222024-07-26T02:34:32.767ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
7232024-07-26T02:34:32.767ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7242024-07-26T02:34:32.785ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
7252024-07-26T02:34:32.785ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7262024-07-26T02:34:32.802ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
7272024-07-26T02:34:32.802ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7282024-07-26T02:34:32.821ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7292024-07-26T02:34:32.821ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7302024-07-26T02:34:32.838ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7312024-07-26T02:34:32.838ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7322024-07-26T02:34:32.856ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7332024-07-26T02:34:32.856ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7342024-07-26T02:34:32.873ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7352024-07-26T02:34:32.873ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7362024-07-26T02:34:32.891ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7372024-07-26T02:34:32.891ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7382024-07-26T02:34:32.909ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
7392024-07-26T02:34:32.909ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
7402024-07-26T02:34:32.910ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
7412024-07-26T02:34:32.910ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
7422024-07-26T02:34:32.941ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
7432024-07-26T02:34:32.941ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
7442024-07-26T02:34:32.941ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
7452024-07-26T02:34:32.941ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
7462024-07-26T02:34:32.941ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
7472024-07-26T02:34:32.941ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c", TimeNotSynchronized), ("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3", TimeNotSynchronized), ("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63", TimeNotSynchronized), ("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6", TimeNotSynchronized), ("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 174385 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52946 req_id = e1d55137-82c1-40e5-a564-077a91d1104e response_code = 500 uri = /omicron-zones
7482024-07-26T02:34:32.941ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "e1d55137-82c1-40e5-a564-077a91d1104e", "content-length": "124", "date": "Fri, 26 Jul 2024 02:34:32 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "e1d55137-82c1-40e5-a564-077a91d1104e" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 96.243026ms sled_address = [fd00:1122:3344:101::1]:12345
7492024-07-26T02:34:33.038ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 25724e7e-f0f3-413d-ab41-b4a33973050b, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 82379658-84e2-487b-bcd4-b30ffc1d9874, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0eb4388e-1e4e-49da-b41b-58746266c965, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bbb96e05-c4c2-4d74-93ab-32168e25b9fd, kind: Service { id: 0eb4388e-1e4e-49da-b41b-58746266c965 }, name: Name("ntp-0eb4388e-1e4e-49da-b41b-58746266c965"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 87])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } } } }, OmicronZoneConfig { id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
7502024-07-26T02:34:33.038ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
7512024-07-26T02:34:33.038ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7522024-07-26T02:34:33.057ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
7532024-07-26T02:34:33.057ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7542024-07-26T02:34:33.075ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
7552024-07-26T02:34:33.075ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7562024-07-26T02:34:33.095ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7572024-07-26T02:34:33.095ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7582024-07-26T02:34:33.113ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7592024-07-26T02:34:33.113ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7602024-07-26T02:34:33.131ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7612024-07-26T02:34:33.131ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7622024-07-26T02:34:33.151ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7632024-07-26T02:34:33.151ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7642024-07-26T02:34:33.168ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7652024-07-26T02:34:33.168ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7662024-07-26T02:34:33.186ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
7672024-07-26T02:34:33.186ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
7682024-07-26T02:34:33.186ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
7692024-07-26T02:34:33.186ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
7702024-07-26T02:34:33.219ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
7712024-07-26T02:34:33.219ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
7722024-07-26T02:34:33.219ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
7732024-07-26T02:34:33.219ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
7742024-07-26T02:34:33.219ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
7752024-07-26T02:34:33.220ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3", TimeNotSynchronized), ("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63", TimeNotSynchronized), ("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c", TimeNotSynchronized), ("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6", TimeNotSynchronized), ("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 181794 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52946 req_id = b77943c8-30e8-4639-8a39-a36ac86c4a7f response_code = 500 uri = /omicron-zones
7762024-07-26T02:34:33.220ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "b77943c8-30e8-4639-8a39-a36ac86c4a7f", "content-length": "124", "date": "Fri, 26 Jul 2024 02:34:33 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "b77943c8-30e8-4639-8a39-a36ac86c4a7f" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 276.568585ms sled_address = [fd00:1122:3344:101::1]:12345
7772024-07-26T02:34:33.496ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 25724e7e-f0f3-413d-ab41-b4a33973050b, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 82379658-84e2-487b-bcd4-b30ffc1d9874, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0eb4388e-1e4e-49da-b41b-58746266c965, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bbb96e05-c4c2-4d74-93ab-32168e25b9fd, kind: Service { id: 0eb4388e-1e4e-49da-b41b-58746266c965 }, name: Name("ntp-0eb4388e-1e4e-49da-b41b-58746266c965"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 87])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } } } }, OmicronZoneConfig { id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
7782024-07-26T02:34:33.497ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
7792024-07-26T02:34:33.497ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7802024-07-26T02:34:33.518ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
7812024-07-26T02:34:33.518ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7822024-07-26T02:34:33.536ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
7832024-07-26T02:34:33.537ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7842024-07-26T02:34:33.554ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7852024-07-26T02:34:33.554ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7862024-07-26T02:34:33.572ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7872024-07-26T02:34:33.572ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7882024-07-26T02:34:33.590ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7892024-07-26T02:34:33.590ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7902024-07-26T02:34:33.608ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7912024-07-26T02:34:33.608ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7922024-07-26T02:34:33.626ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7932024-07-26T02:34:33.626ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7942024-07-26T02:34:33.644ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
7952024-07-26T02:34:33.644ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
7962024-07-26T02:34:33.645ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
7972024-07-26T02:34:33.645ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
7982024-07-26T02:34:33.676ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
7992024-07-26T02:34:33.676ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
8002024-07-26T02:34:33.676ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
8012024-07-26T02:34:33.676ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
8022024-07-26T02:34:33.677ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
8032024-07-26T02:34:33.677ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8", TimeNotSynchronized), ("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c", TimeNotSynchronized), ("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3", TimeNotSynchronized), ("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6", TimeNotSynchronized), ("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 180661 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52946 req_id = cf5af7c3-4f2d-4e10-8d60-be59d292e426 response_code = 500 uri = /omicron-zones
8042024-07-26T02:34:33.677ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "cf5af7c3-4f2d-4e10-8d60-be59d292e426", "content-length": "124", "date": "Fri, 26 Jul 2024 02:34:33 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "cf5af7c3-4f2d-4e10-8d60-be59d292e426" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 266.679497ms sled_address = [fd00:1122:3344:101::1]:12345
8052024-07-26T02:34:33.943ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 25724e7e-f0f3-413d-ab41-b4a33973050b, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 82379658-84e2-487b-bcd4-b30ffc1d9874, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0eb4388e-1e4e-49da-b41b-58746266c965, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bbb96e05-c4c2-4d74-93ab-32168e25b9fd, kind: Service { id: 0eb4388e-1e4e-49da-b41b-58746266c965 }, name: Name("ntp-0eb4388e-1e4e-49da-b41b-58746266c965"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 87])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } } } }, OmicronZoneConfig { id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
8062024-07-26T02:34:33.944ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
8072024-07-26T02:34:33.944ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8082024-07-26T02:34:33.963ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
8092024-07-26T02:34:33.963ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8102024-07-26T02:34:33.980ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
8112024-07-26T02:34:33.980ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8122024-07-26T02:34:33.996ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8132024-07-26T02:34:33.996ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8142024-07-26T02:34:34.014ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8152024-07-26T02:34:34.014ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8162024-07-26T02:34:34.030ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8172024-07-26T02:34:34.030ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8182024-07-26T02:34:34.049ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8192024-07-26T02:34:34.050ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8202024-07-26T02:34:34.068ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8212024-07-26T02:34:34.068ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8222024-07-26T02:34:34.083ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
8232024-07-26T02:34:34.083ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
8242024-07-26T02:34:34.084ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
8252024-07-26T02:34:34.084ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
8262024-07-26T02:34:34.115ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
8272024-07-26T02:34:34.115ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
8282024-07-26T02:34:34.115ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
8292024-07-26T02:34:34.115ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
8302024-07-26T02:34:34.115ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
8312024-07-26T02:34:34.115ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6", TimeNotSynchronized), ("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c", TimeNotSynchronized), ("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8", TimeNotSynchronized), ("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3", TimeNotSynchronized), ("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 171915 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52946 req_id = 1c2a28d7-84a5-4b4c-85e3-31de2dc00d9a response_code = 500 uri = /omicron-zones
8322024-07-26T02:34:34.115ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "1c2a28d7-84a5-4b4c-85e3-31de2dc00d9a", "content-length": "124", "date": "Fri, 26 Jul 2024 02:34:33 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "1c2a28d7-84a5-4b4c-85e3-31de2dc00d9a" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 292.151585ms sled_address = [fd00:1122:3344:101::1]:12345
8332024-07-26T02:34:34.408ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 25724e7e-f0f3-413d-ab41-b4a33973050b, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 82379658-84e2-487b-bcd4-b30ffc1d9874, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0eb4388e-1e4e-49da-b41b-58746266c965, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bbb96e05-c4c2-4d74-93ab-32168e25b9fd, kind: Service { id: 0eb4388e-1e4e-49da-b41b-58746266c965 }, name: Name("ntp-0eb4388e-1e4e-49da-b41b-58746266c965"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 87])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } } } }, OmicronZoneConfig { id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
8342024-07-26T02:34:34.408ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
8352024-07-26T02:34:34.408ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8362024-07-26T02:34:34.428ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
8372024-07-26T02:34:34.428ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8382024-07-26T02:34:34.447ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
8392024-07-26T02:34:34.447ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8402024-07-26T02:34:34.465ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8412024-07-26T02:34:34.465ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8422024-07-26T02:34:34.482ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8432024-07-26T02:34:34.482ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8442024-07-26T02:34:34.500ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8452024-07-26T02:34:34.500ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8462024-07-26T02:34:34.517ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8472024-07-26T02:34:34.517ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8482024-07-26T02:34:34.535ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8492024-07-26T02:34:34.535ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8502024-07-26T02:34:34.552ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
8512024-07-26T02:34:34.552ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
8522024-07-26T02:34:34.552ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
8532024-07-26T02:34:34.552ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
8542024-07-26T02:34:34.584ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
8552024-07-26T02:34:34.584ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
8562024-07-26T02:34:34.584ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
8572024-07-26T02:34:34.584ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
8582024-07-26T02:34:34.584ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
8592024-07-26T02:34:34.584ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8", TimeNotSynchronized), ("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6", TimeNotSynchronized), ("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c", TimeNotSynchronized), ("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63", TimeNotSynchronized), ("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 176178 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52946 req_id = 6c703f39-bcbc-4e1f-acbc-2b7b6946149b response_code = 500 uri = /omicron-zones
8602024-07-26T02:34:34.584ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "6c703f39-bcbc-4e1f-acbc-2b7b6946149b", "content-length": "124", "date": "Fri, 26 Jul 2024 02:34:34 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "6c703f39-bcbc-4e1f-acbc-2b7b6946149b" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 286.264619ms sled_address = [fd00:1122:3344:101::1]:12345
8612024-07-26T02:34:34.813ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 306.321315ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
8622024-07-26T02:34:34.871ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 25724e7e-f0f3-413d-ab41-b4a33973050b, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 82379658-84e2-487b-bcd4-b30ffc1d9874, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0eb4388e-1e4e-49da-b41b-58746266c965, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bbb96e05-c4c2-4d74-93ab-32168e25b9fd, kind: Service { id: 0eb4388e-1e4e-49da-b41b-58746266c965 }, name: Name("ntp-0eb4388e-1e4e-49da-b41b-58746266c965"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 87])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } } } }, OmicronZoneConfig { id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
8632024-07-26T02:34:34.871ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
8642024-07-26T02:34:34.871ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8652024-07-26T02:34:34.890ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
8662024-07-26T02:34:34.890ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8672024-07-26T02:34:34.907ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
8682024-07-26T02:34:34.907ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8692024-07-26T02:34:34.924ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8702024-07-26T02:34:34.925ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8712024-07-26T02:34:34.942ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8722024-07-26T02:34:34.942ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8732024-07-26T02:34:34.961ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8742024-07-26T02:34:34.961ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8752024-07-26T02:34:34.979ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8762024-07-26T02:34:34.979ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8772024-07-26T02:34:34.996ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8782024-07-26T02:34:34.996ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8792024-07-26T02:34:35.014ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
8802024-07-26T02:34:35.014ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
8812024-07-26T02:34:35.014ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
8822024-07-26T02:34:35.014ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
8832024-07-26T02:34:35.047ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
8842024-07-26T02:34:35.047ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
8852024-07-26T02:34:35.047ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
8862024-07-26T02:34:35.047ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
8872024-07-26T02:34:35.047ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
8882024-07-26T02:34:35.047ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c", TimeNotSynchronized), ("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8", TimeNotSynchronized), ("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63", TimeNotSynchronized), ("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6", TimeNotSynchronized), ("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 176917 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52946 req_id = 04e9b92a-cfff-4880-b849-282700118d55 response_code = 500 uri = /omicron-zones
8892024-07-26T02:34:35.047ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "04e9b92a-cfff-4880-b849-282700118d55", "content-length": "124", "date": "Fri, 26 Jul 2024 02:34:34 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "04e9b92a-cfff-4880-b849-282700118d55" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 384.254657ms sled_address = [fd00:1122:3344:101::1]:12345
8902024-07-26T02:34:35.432ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 25724e7e-f0f3-413d-ab41-b4a33973050b, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 82379658-84e2-487b-bcd4-b30ffc1d9874, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0eb4388e-1e4e-49da-b41b-58746266c965, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bbb96e05-c4c2-4d74-93ab-32168e25b9fd, kind: Service { id: 0eb4388e-1e4e-49da-b41b-58746266c965 }, name: Name("ntp-0eb4388e-1e4e-49da-b41b-58746266c965"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 87])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } } } }, OmicronZoneConfig { id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
8912024-07-26T02:34:35.432ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
8922024-07-26T02:34:35.432ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8932024-07-26T02:34:35.451ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
8942024-07-26T02:34:35.451ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8952024-07-26T02:34:35.469ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
8962024-07-26T02:34:35.469ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8972024-07-26T02:34:35.486ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8982024-07-26T02:34:35.486ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8992024-07-26T02:34:35.504ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9002024-07-26T02:34:35.504ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9012024-07-26T02:34:35.522ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9022024-07-26T02:34:35.522ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9032024-07-26T02:34:35.540ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9042024-07-26T02:34:35.540ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9052024-07-26T02:34:35.558ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9062024-07-26T02:34:35.558ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9072024-07-26T02:34:35.575ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
9082024-07-26T02:34:35.576ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
9092024-07-26T02:34:35.576ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
9102024-07-26T02:34:35.576ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
9112024-07-26T02:34:35.608ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
9122024-07-26T02:34:35.608ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
9132024-07-26T02:34:35.608ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
9142024-07-26T02:34:35.608ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
9152024-07-26T02:34:35.608ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
9162024-07-26T02:34:35.608ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6", TimeNotSynchronized), ("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63", TimeNotSynchronized), ("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8", TimeNotSynchronized), ("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c", TimeNotSynchronized), ("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 176595 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52946 req_id = 524e999d-7a1c-47c1-b2f3-bdeb60771e81 response_code = 500 uri = /omicron-zones
9172024-07-26T02:34:35.608ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "524e999d-7a1c-47c1-b2f3-bdeb60771e81", "content-length": "124", "date": "Fri, 26 Jul 2024 02:34:35 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "524e999d-7a1c-47c1-b2f3-bdeb60771e81" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 682.481924ms sled_address = [fd00:1122:3344:101::1]:12345
9182024-07-26T02:34:36.290ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 25724e7e-f0f3-413d-ab41-b4a33973050b, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 82379658-84e2-487b-bcd4-b30ffc1d9874, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0eb4388e-1e4e-49da-b41b-58746266c965, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bbb96e05-c4c2-4d74-93ab-32168e25b9fd, kind: Service { id: 0eb4388e-1e4e-49da-b41b-58746266c965 }, name: Name("ntp-0eb4388e-1e4e-49da-b41b-58746266c965"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 87])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } } } }, OmicronZoneConfig { id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
9192024-07-26T02:34:36.290ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
9202024-07-26T02:34:36.290ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9212024-07-26T02:34:36.310ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
9222024-07-26T02:34:36.310ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9232024-07-26T02:34:36.328ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
9242024-07-26T02:34:36.328ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9252024-07-26T02:34:36.346ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9262024-07-26T02:34:36.346ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9272024-07-26T02:34:36.364ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9282024-07-26T02:34:36.364ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9292024-07-26T02:34:36.381ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9302024-07-26T02:34:36.381ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9312024-07-26T02:34:36.399ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9322024-07-26T02:34:36.399ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9332024-07-26T02:34:36.417ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9342024-07-26T02:34:36.417ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9352024-07-26T02:34:36.435ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
9362024-07-26T02:34:36.435ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
9372024-07-26T02:34:36.435ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
9382024-07-26T02:34:36.436ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
9392024-07-26T02:34:36.466ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
9402024-07-26T02:34:36.466ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
9412024-07-26T02:34:36.466ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
9422024-07-26T02:34:36.466ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
9432024-07-26T02:34:36.466ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
9442024-07-26T02:34:36.466ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6", TimeNotSynchronized), ("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3", TimeNotSynchronized), ("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63", TimeNotSynchronized), ("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8", TimeNotSynchronized), ("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 176889 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52946 req_id = 66382181-ca96-469f-9dbf-e356307c44ac response_code = 500 uri = /omicron-zones
9452024-07-26T02:34:36.467ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "66382181-ca96-469f-9dbf-e356307c44ac", "content-length": "124", "date": "Fri, 26 Jul 2024 02:34:36 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "66382181-ca96-469f-9dbf-e356307c44ac" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 496.701775ms sled_address = [fd00:1122:3344:101::1]:12345
9462024-07-26T02:34:36.963ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 25724e7e-f0f3-413d-ab41-b4a33973050b, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 82379658-84e2-487b-bcd4-b30ffc1d9874, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0eb4388e-1e4e-49da-b41b-58746266c965, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bbb96e05-c4c2-4d74-93ab-32168e25b9fd, kind: Service { id: 0eb4388e-1e4e-49da-b41b-58746266c965 }, name: Name("ntp-0eb4388e-1e4e-49da-b41b-58746266c965"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 87])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } } } }, OmicronZoneConfig { id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
9472024-07-26T02:34:36.964ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
9482024-07-26T02:34:36.964ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9492024-07-26T02:34:36.985ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
9502024-07-26T02:34:36.985ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9512024-07-26T02:34:37.004ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
9522024-07-26T02:34:37.004ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9532024-07-26T02:34:37.023ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9542024-07-26T02:34:37.023ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9552024-07-26T02:34:37.040ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9562024-07-26T02:34:37.040ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9572024-07-26T02:34:37.057ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9582024-07-26T02:34:37.057ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9592024-07-26T02:34:37.076ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9602024-07-26T02:34:37.076ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9612024-07-26T02:34:37.094ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9622024-07-26T02:34:37.094ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9632024-07-26T02:34:37.111ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
9642024-07-26T02:34:37.111ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
9652024-07-26T02:34:37.111ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
9662024-07-26T02:34:37.111ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
9672024-07-26T02:34:37.142ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
9682024-07-26T02:34:37.143ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
9692024-07-26T02:34:37.143ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
9702024-07-26T02:34:37.143ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
9712024-07-26T02:34:37.143ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
9722024-07-26T02:34:37.143ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c", TimeNotSynchronized), ("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63", TimeNotSynchronized), ("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8", TimeNotSynchronized), ("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6", TimeNotSynchronized), ("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 180161 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52946 req_id = 132101db-1537-4db0-89f9-e175fb706594 response_code = 500 uri = /omicron-zones
9732024-07-26T02:34:37.143ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "132101db-1537-4db0-89f9-e175fb706594", "content-length": "124", "date": "Fri, 26 Jul 2024 02:34:36 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "132101db-1537-4db0-89f9-e175fb706594" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 395.408957ms sled_address = [fd00:1122:3344:101::1]:12345
9742024-07-26T02:34:37.538ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 25724e7e-f0f3-413d-ab41-b4a33973050b, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 82379658-84e2-487b-bcd4-b30ffc1d9874, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0eb4388e-1e4e-49da-b41b-58746266c965, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bbb96e05-c4c2-4d74-93ab-32168e25b9fd, kind: Service { id: 0eb4388e-1e4e-49da-b41b-58746266c965 }, name: Name("ntp-0eb4388e-1e4e-49da-b41b-58746266c965"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 87])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } } } }, OmicronZoneConfig { id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
9752024-07-26T02:34:37.538ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
9762024-07-26T02:34:37.538ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9772024-07-26T02:34:37.559ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
9782024-07-26T02:34:37.559ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9792024-07-26T02:34:37.577ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
9802024-07-26T02:34:37.577ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9812024-07-26T02:34:37.594ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9822024-07-26T02:34:37.594ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9832024-07-26T02:34:37.613ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9842024-07-26T02:34:37.613ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9852024-07-26T02:34:37.631ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9862024-07-26T02:34:37.631ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9872024-07-26T02:34:37.649ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9882024-07-26T02:34:37.649ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9892024-07-26T02:34:37.665ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9902024-07-26T02:34:37.665ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9912024-07-26T02:34:37.684ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
9922024-07-26T02:34:37.684ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
9932024-07-26T02:34:37.684ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
9942024-07-26T02:34:37.685ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
9952024-07-26T02:34:37.715ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
9962024-07-26T02:34:37.716ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
9972024-07-26T02:34:37.716ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
9982024-07-26T02:34:37.716ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
9992024-07-26T02:34:37.716ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
10002024-07-26T02:34:37.716ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c", TimeNotSynchronized), ("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63", TimeNotSynchronized), ("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3", TimeNotSynchronized), ("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8", TimeNotSynchronized), ("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 178210 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52946 req_id = bd178f74-0d03-43cb-960c-5e5d4a2f5b47 response_code = 500 uri = /omicron-zones
10012024-07-26T02:34:37.716ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "bd178f74-0d03-43cb-960c-5e5d4a2f5b47", "content-length": "124", "date": "Fri, 26 Jul 2024 02:34:36 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "bd178f74-0d03-43cb-960c-5e5d4a2f5b47" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 548.273299ms sled_address = [fd00:1122:3344:101::1]:12345
10022024-07-26T02:34:38.263ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 25724e7e-f0f3-413d-ab41-b4a33973050b, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 82379658-84e2-487b-bcd4-b30ffc1d9874, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0eb4388e-1e4e-49da-b41b-58746266c965, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bbb96e05-c4c2-4d74-93ab-32168e25b9fd, kind: Service { id: 0eb4388e-1e4e-49da-b41b-58746266c965 }, name: Name("ntp-0eb4388e-1e4e-49da-b41b-58746266c965"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 87])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } } } }, OmicronZoneConfig { id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
10032024-07-26T02:34:38.263ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
10042024-07-26T02:34:38.263ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10052024-07-26T02:34:38.282ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
10062024-07-26T02:34:38.282ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10072024-07-26T02:34:38.300ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
10082024-07-26T02:34:38.300ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10092024-07-26T02:34:38.317ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10102024-07-26T02:34:38.318ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10112024-07-26T02:34:38.335ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10122024-07-26T02:34:38.335ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10132024-07-26T02:34:38.353ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10142024-07-26T02:34:38.353ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10152024-07-26T02:34:38.371ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10162024-07-26T02:34:38.371ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10172024-07-26T02:34:38.389ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10182024-07-26T02:34:38.389ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10192024-07-26T02:34:38.408ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
10202024-07-26T02:34:38.408ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
10212024-07-26T02:34:38.408ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
10222024-07-26T02:34:38.408ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
10232024-07-26T02:34:38.441ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
10242024-07-26T02:34:38.441ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
10252024-07-26T02:34:38.441ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
10262024-07-26T02:34:38.441ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
10272024-07-26T02:34:38.441ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
10282024-07-26T02:34:38.441ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8", TimeNotSynchronized), ("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6", TimeNotSynchronized), ("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63", TimeNotSynchronized), ("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3", TimeNotSynchronized), ("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 178586 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52946 req_id = 1a09820d-8c69-486b-a91e-35cf73efb966 response_code = 500 uri = /omicron-zones
10292024-07-26T02:34:38.441ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "1a09820d-8c69-486b-a91e-35cf73efb966", "content-length": "124", "date": "Fri, 26 Jul 2024 02:34:37 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "1a09820d-8c69-486b-a91e-35cf73efb966" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 787.22934ms sled_address = [fd00:1122:3344:101::1]:12345
10302024-07-26T02:34:39.227ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 25724e7e-f0f3-413d-ab41-b4a33973050b, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 82379658-84e2-487b-bcd4-b30ffc1d9874, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0eb4388e-1e4e-49da-b41b-58746266c965, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bbb96e05-c4c2-4d74-93ab-32168e25b9fd, kind: Service { id: 0eb4388e-1e4e-49da-b41b-58746266c965 }, name: Name("ntp-0eb4388e-1e4e-49da-b41b-58746266c965"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 87])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } } } }, OmicronZoneConfig { id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
10312024-07-26T02:34:39.228ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
10322024-07-26T02:34:39.228ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10332024-07-26T02:34:39.248ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
10342024-07-26T02:34:39.248ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10352024-07-26T02:34:39.267ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
10362024-07-26T02:34:39.267ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10372024-07-26T02:34:39.285ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10382024-07-26T02:34:39.286ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10392024-07-26T02:34:39.304ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10402024-07-26T02:34:39.304ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10412024-07-26T02:34:39.322ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10422024-07-26T02:34:39.322ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10432024-07-26T02:34:39.339ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10442024-07-26T02:34:39.339ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10452024-07-26T02:34:39.357ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10462024-07-26T02:34:39.357ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10472024-07-26T02:34:39.376ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
10482024-07-26T02:34:39.376ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
10492024-07-26T02:34:39.376ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
10502024-07-26T02:34:39.377ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
10512024-07-26T02:34:39.409ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
10522024-07-26T02:34:39.409ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
10532024-07-26T02:34:39.409ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
10542024-07-26T02:34:39.409ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
10552024-07-26T02:34:39.409ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
10562024-07-26T02:34:39.409ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c", TimeNotSynchronized), ("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3", TimeNotSynchronized), ("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8", TimeNotSynchronized), ("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6", TimeNotSynchronized), ("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 181598 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52946 req_id = 724dbf5a-5454-4442-86ea-3c4ae0e374d9 response_code = 500 uri = /omicron-zones
10572024-07-26T02:34:39.409ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "724dbf5a-5454-4442-86ea-3c4ae0e374d9", "content-length": "124", "date": "Fri, 26 Jul 2024 02:34:39 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "724dbf5a-5454-4442-86ea-3c4ae0e374d9" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 788.904792ms sled_address = [fd00:1122:3344:101::1]:12345
10582024-07-26T02:34:40.197ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 25724e7e-f0f3-413d-ab41-b4a33973050b, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 82379658-84e2-487b-bcd4-b30ffc1d9874, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0eb4388e-1e4e-49da-b41b-58746266c965, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bbb96e05-c4c2-4d74-93ab-32168e25b9fd, kind: Service { id: 0eb4388e-1e4e-49da-b41b-58746266c965 }, name: Name("ntp-0eb4388e-1e4e-49da-b41b-58746266c965"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 87])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } } } }, OmicronZoneConfig { id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
10592024-07-26T02:34:40.199ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
10602024-07-26T02:34:40.199ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10612024-07-26T02:34:40.219ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
10622024-07-26T02:34:40.219ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10632024-07-26T02:34:40.237ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
10642024-07-26T02:34:40.237ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10652024-07-26T02:34:40.255ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10662024-07-26T02:34:40.255ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10672024-07-26T02:34:40.272ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10682024-07-26T02:34:40.272ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10692024-07-26T02:34:40.289ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10702024-07-26T02:34:40.289ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10712024-07-26T02:34:40.306ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10722024-07-26T02:34:40.306ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10732024-07-26T02:34:40.325ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10742024-07-26T02:34:40.325ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10752024-07-26T02:34:40.341ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
10762024-07-26T02:34:40.341ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
10772024-07-26T02:34:40.342ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
10782024-07-26T02:34:40.342ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
10792024-07-26T02:34:40.374ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
10802024-07-26T02:34:40.374ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
10812024-07-26T02:34:40.374ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
10822024-07-26T02:34:40.374ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
10832024-07-26T02:34:40.374ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
10842024-07-26T02:34:40.374ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6", TimeNotSynchronized), ("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3", TimeNotSynchronized), ("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c", TimeNotSynchronized), ("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8", TimeNotSynchronized), ("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 176906 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52946 req_id = b2d9486f-6f96-4ce5-884c-f933786f627b response_code = 500 uri = /omicron-zones
10852024-07-26T02:34:40.374ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "b2d9486f-6f96-4ce5-884c-f933786f627b", "content-length": "124", "date": "Fri, 26 Jul 2024 02:34:40 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "b2d9486f-6f96-4ce5-884c-f933786f627b" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 1.446892908s sled_address = [fd00:1122:3344:101::1]:12345
10862024-07-26T02:34:41.818ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 25724e7e-f0f3-413d-ab41-b4a33973050b, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 82379658-84e2-487b-bcd4-b30ffc1d9874, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0eb4388e-1e4e-49da-b41b-58746266c965, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bbb96e05-c4c2-4d74-93ab-32168e25b9fd, kind: Service { id: 0eb4388e-1e4e-49da-b41b-58746266c965 }, name: Name("ntp-0eb4388e-1e4e-49da-b41b-58746266c965"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 87])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } } } }, OmicronZoneConfig { id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
10872024-07-26T02:34:41.819ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
10882024-07-26T02:34:41.819ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10892024-07-26T02:34:41.840ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
10902024-07-26T02:34:41.841ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10912024-07-26T02:34:41.861ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
10922024-07-26T02:34:41.861ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10932024-07-26T02:34:41.880ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10942024-07-26T02:34:41.880ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10952024-07-26T02:34:41.899ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10962024-07-26T02:34:41.899ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10972024-07-26T02:34:41.919ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10982024-07-26T02:34:41.919ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10992024-07-26T02:34:41.937ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11002024-07-26T02:34:41.937ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11012024-07-26T02:34:41.956ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11022024-07-26T02:34:41.956ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11032024-07-26T02:34:41.976ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
11042024-07-26T02:34:41.976ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
11052024-07-26T02:34:41.976ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
11062024-07-26T02:34:41.976ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
11072024-07-26T02:34:42.009ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
11082024-07-26T02:34:42.009ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
11092024-07-26T02:34:42.009ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
11102024-07-26T02:34:42.009ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
11112024-07-26T02:34:42.009ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
11122024-07-26T02:34:42.009ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c", TimeNotSynchronized), ("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3", TimeNotSynchronized), ("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8", TimeNotSynchronized), ("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6", TimeNotSynchronized), ("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 191542 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52946 req_id = d210d2ce-b3f1-4e19-81ca-5ee5a1c63bca response_code = 500 uri = /omicron-zones
11132024-07-26T02:34:42.009ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "d210d2ce-b3f1-4e19-81ca-5ee5a1c63bca", "content-length": "124", "date": "Fri, 26 Jul 2024 02:34:41 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "d210d2ce-b3f1-4e19-81ca-5ee5a1c63bca" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 2.42801443s sled_address = [fd00:1122:3344:101::1]:12345
11142024-07-26T02:34:44.432ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 25724e7e-f0f3-413d-ab41-b4a33973050b, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 82379658-84e2-487b-bcd4-b30ffc1d9874, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0eb4388e-1e4e-49da-b41b-58746266c965, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bbb96e05-c4c2-4d74-93ab-32168e25b9fd, kind: Service { id: 0eb4388e-1e4e-49da-b41b-58746266c965 }, name: Name("ntp-0eb4388e-1e4e-49da-b41b-58746266c965"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 87])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } } } }, OmicronZoneConfig { id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
11152024-07-26T02:34:44.433ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
11162024-07-26T02:34:44.433ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11172024-07-26T02:34:44.452ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
11182024-07-26T02:34:44.452ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11192024-07-26T02:34:44.472ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
11202024-07-26T02:34:44.472ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11212024-07-26T02:34:44.490ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11222024-07-26T02:34:44.490ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11232024-07-26T02:34:44.509ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11242024-07-26T02:34:44.509ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11252024-07-26T02:34:44.528ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11262024-07-26T02:34:44.528ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11272024-07-26T02:34:44.546ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11282024-07-26T02:34:44.546ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11292024-07-26T02:34:44.564ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11302024-07-26T02:34:44.564ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11312024-07-26T02:34:44.582ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
11322024-07-26T02:34:44.582ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
11332024-07-26T02:34:44.582ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
11342024-07-26T02:34:44.582ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
11352024-07-26T02:34:44.616ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
11362024-07-26T02:34:44.616ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
11372024-07-26T02:34:44.616ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
11382024-07-26T02:34:44.616ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
11392024-07-26T02:34:44.616ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
11402024-07-26T02:34:44.616ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3", TimeNotSynchronized), ("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6", TimeNotSynchronized), ("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8", TimeNotSynchronized), ("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c", TimeNotSynchronized), ("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 184230 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52946 req_id = 7b07c8fb-cae8-4a15-8fa1-3297799e8014 response_code = 500 uri = /omicron-zones
11412024-07-26T02:34:44.616ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "7b07c8fb-cae8-4a15-8fa1-3297799e8014", "content-length": "124", "date": "Fri, 26 Jul 2024 02:34:44 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "7b07c8fb-cae8-4a15-8fa1-3297799e8014" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 3.202822289s sled_address = [fd00:1122:3344:101::1]:12345
11422024-07-26T02:34:46.393ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:42950
11432024-07-26T02:34:46.393ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 79 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:42950 req_id = 028bc0ed-f281-4076-8f0e-f5a09f4d5165 response_code = 200 uri = /baseboard
11442024-07-26T02:34:47.811ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 25724e7e-f0f3-413d-ab41-b4a33973050b, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 82379658-84e2-487b-bcd4-b30ffc1d9874, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0eb4388e-1e4e-49da-b41b-58746266c965, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bbb96e05-c4c2-4d74-93ab-32168e25b9fd, kind: Service { id: 0eb4388e-1e4e-49da-b41b-58746266c965 }, name: Name("ntp-0eb4388e-1e4e-49da-b41b-58746266c965"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 87])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } } } }, OmicronZoneConfig { id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
11452024-07-26T02:34:47.811ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
11462024-07-26T02:34:47.811ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11472024-07-26T02:34:47.833ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
11482024-07-26T02:34:47.833ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11492024-07-26T02:34:47.851ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
11502024-07-26T02:34:47.851ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11512024-07-26T02:34:47.869ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11522024-07-26T02:34:47.869ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11532024-07-26T02:34:47.886ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11542024-07-26T02:34:47.886ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11552024-07-26T02:34:47.903ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11562024-07-26T02:34:47.903ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11572024-07-26T02:34:47.920ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11582024-07-26T02:34:47.920ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11592024-07-26T02:34:47.937ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11602024-07-26T02:34:47.937ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11612024-07-26T02:34:47.954ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
11622024-07-26T02:34:47.954ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
11632024-07-26T02:34:47.954ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
11642024-07-26T02:34:47.954ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
11652024-07-26T02:34:47.985ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
11662024-07-26T02:34:47.985ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
11672024-07-26T02:34:47.985ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
11682024-07-26T02:34:47.985ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
11692024-07-26T02:34:47.985ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
11702024-07-26T02:34:47.985ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6", TimeNotSynchronized), ("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3", TimeNotSynchronized), ("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63", TimeNotSynchronized), ("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8", TimeNotSynchronized), ("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 174720 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52946 req_id = e4c5525e-b052-48a0-a0ed-08c95c1f0d71 response_code = 500 uri = /omicron-zones
11712024-07-26T02:34:47.986ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "e4c5525e-b052-48a0-a0ed-08c95c1f0d71", "content-length": "124", "date": "Fri, 26 Jul 2024 02:34:47 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "e4c5525e-b052-48a0-a0ed-08c95c1f0d71" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 2.094100738s sled_address = [fd00:1122:3344:101::1]:12345
11722024-07-26T02:34:50.075ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 25724e7e-f0f3-413d-ab41-b4a33973050b, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 82379658-84e2-487b-bcd4-b30ffc1d9874, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0eb4388e-1e4e-49da-b41b-58746266c965, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bbb96e05-c4c2-4d74-93ab-32168e25b9fd, kind: Service { id: 0eb4388e-1e4e-49da-b41b-58746266c965 }, name: Name("ntp-0eb4388e-1e4e-49da-b41b-58746266c965"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 87])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } } } }, OmicronZoneConfig { id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
11732024-07-26T02:34:50.075ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
11742024-07-26T02:34:50.075ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11752024-07-26T02:34:50.096ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
11762024-07-26T02:34:50.096ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11772024-07-26T02:34:50.113ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
11782024-07-26T02:34:50.113ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11792024-07-26T02:34:50.118ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 368.218682ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
11802024-07-26T02:34:50.132ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11812024-07-26T02:34:50.132ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11822024-07-26T02:34:50.150ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11832024-07-26T02:34:50.150ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11842024-07-26T02:34:50.167ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11852024-07-26T02:34:50.167ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11862024-07-26T02:34:50.185ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11872024-07-26T02:34:50.185ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11882024-07-26T02:34:50.203ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11892024-07-26T02:34:50.203ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11902024-07-26T02:34:50.220ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
11912024-07-26T02:34:50.221ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
11922024-07-26T02:34:50.221ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
11932024-07-26T02:34:50.221ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
11942024-07-26T02:34:50.253ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
11952024-07-26T02:34:50.253ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
11962024-07-26T02:34:50.253ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
11972024-07-26T02:34:50.253ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
11982024-07-26T02:34:50.253ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
11992024-07-26T02:34:50.253ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3", TimeNotSynchronized), ("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6", TimeNotSynchronized), ("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c", TimeNotSynchronized), ("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63", TimeNotSynchronized), ("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 178454 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52946 req_id = bae4f230-491e-47fb-b101-29bb8a854b9d response_code = 500 uri = /omicron-zones
12002024-07-26T02:34:50.253ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "bae4f230-491e-47fb-b101-29bb8a854b9d", "content-length": "124", "date": "Fri, 26 Jul 2024 02:34:50 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "bae4f230-491e-47fb-b101-29bb8a854b9d" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 3.990933601s sled_address = [fd00:1122:3344:101::1]:12345
12012024-07-26T02:34:54.234ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 25724e7e-f0f3-413d-ab41-b4a33973050b, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 82379658-84e2-487b-bcd4-b30ffc1d9874, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0eb4388e-1e4e-49da-b41b-58746266c965, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bbb96e05-c4c2-4d74-93ab-32168e25b9fd, kind: Service { id: 0eb4388e-1e4e-49da-b41b-58746266c965 }, name: Name("ntp-0eb4388e-1e4e-49da-b41b-58746266c965"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 87])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } } } }, OmicronZoneConfig { id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
12022024-07-26T02:34:54.235ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
12032024-07-26T02:34:54.235ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12042024-07-26T02:34:54.258ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
12052024-07-26T02:34:54.258ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12062024-07-26T02:34:54.276ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
12072024-07-26T02:34:54.277ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12082024-07-26T02:34:54.295ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12092024-07-26T02:34:54.295ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12102024-07-26T02:34:54.314ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12112024-07-26T02:34:54.314ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12122024-07-26T02:34:54.331ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12132024-07-26T02:34:54.331ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12142024-07-26T02:34:54.349ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12152024-07-26T02:34:54.349ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12162024-07-26T02:34:54.367ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12172024-07-26T02:34:54.367ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12182024-07-26T02:34:54.384ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
12192024-07-26T02:34:54.384ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
12202024-07-26T02:34:54.385ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
12212024-07-26T02:34:54.385ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
12222024-07-26T02:34:54.416ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
12232024-07-26T02:34:54.416ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
12242024-07-26T02:34:54.416ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
12252024-07-26T02:34:54.416ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
12262024-07-26T02:34:54.416ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
12272024-07-26T02:34:54.416ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3", TimeNotSynchronized), ("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6", TimeNotSynchronized), ("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c", TimeNotSynchronized), ("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63", TimeNotSynchronized), ("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 181874 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52946 req_id = e75ed8d5-0f05-4d4b-b18a-bc7d511fb64b response_code = 500 uri = /omicron-zones
12282024-07-26T02:34:54.416ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "e75ed8d5-0f05-4d4b-b18a-bc7d511fb64b", "content-length": "124", "date": "Fri, 26 Jul 2024 02:34:54 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "e75ed8d5-0f05-4d4b-b18a-bc7d511fb64b" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 4.886456722s sled_address = [fd00:1122:3344:101::1]:12345
12292024-07-26T02:34:59.291ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 25724e7e-f0f3-413d-ab41-b4a33973050b, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 82379658-84e2-487b-bcd4-b30ffc1d9874, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0eb4388e-1e4e-49da-b41b-58746266c965, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bbb96e05-c4c2-4d74-93ab-32168e25b9fd, kind: Service { id: 0eb4388e-1e4e-49da-b41b-58746266c965 }, name: Name("ntp-0eb4388e-1e4e-49da-b41b-58746266c965"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 87])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } } } }, OmicronZoneConfig { id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
12302024-07-26T02:34:59.292ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
12312024-07-26T02:34:59.292ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12322024-07-26T02:34:59.313ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
12332024-07-26T02:34:59.313ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12342024-07-26T02:34:59.331ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
12352024-07-26T02:34:59.332ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12362024-07-26T02:34:59.349ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12372024-07-26T02:34:59.349ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12382024-07-26T02:34:59.367ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12392024-07-26T02:34:59.367ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12402024-07-26T02:34:59.385ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12412024-07-26T02:34:59.386ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12422024-07-26T02:34:59.403ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12432024-07-26T02:34:59.404ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12442024-07-26T02:34:59.421ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12452024-07-26T02:34:59.421ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12462024-07-26T02:34:59.438ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
12472024-07-26T02:34:59.438ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
12482024-07-26T02:34:59.439ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
12492024-07-26T02:34:59.439ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
12502024-07-26T02:34:59.470ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
12512024-07-26T02:34:59.470ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
12522024-07-26T02:34:59.470ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
12532024-07-26T02:34:59.470ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
12542024-07-26T02:34:59.470ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
12552024-07-26T02:34:59.470ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8", TimeNotSynchronized), ("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c", TimeNotSynchronized), ("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6", TimeNotSynchronized), ("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3", TimeNotSynchronized), ("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 179126 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52946 req_id = 981bb968-8968-4885-9f36-6c8a37cd7886 response_code = 500 uri = /omicron-zones
12562024-07-26T02:34:59.470ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "981bb968-8968-4885-9f36-6c8a37cd7886", "content-length": "124", "date": "Fri, 26 Jul 2024 02:34:59 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "981bb968-8968-4885-9f36-6c8a37cd7886" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 6.526941977s sled_address = [fd00:1122:3344:101::1]:12345
12572024-07-26T02:35:05.482ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 1.391043226s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
12582024-07-26T02:35:05.981ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 25724e7e-f0f3-413d-ab41-b4a33973050b, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 82379658-84e2-487b-bcd4-b30ffc1d9874, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0eb4388e-1e4e-49da-b41b-58746266c965, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bbb96e05-c4c2-4d74-93ab-32168e25b9fd, kind: Service { id: 0eb4388e-1e4e-49da-b41b-58746266c965 }, name: Name("ntp-0eb4388e-1e4e-49da-b41b-58746266c965"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 87])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } } } }, OmicronZoneConfig { id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
12592024-07-26T02:35:05.981ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
12602024-07-26T02:35:05.981ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12612024-07-26T02:35:06.005ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
12622024-07-26T02:35:06.005ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12632024-07-26T02:35:06.023ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
12642024-07-26T02:35:06.023ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12652024-07-26T02:35:06.040ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12662024-07-26T02:35:06.040ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12672024-07-26T02:35:06.060ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12682024-07-26T02:35:06.060ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12692024-07-26T02:35:06.079ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12702024-07-26T02:35:06.079ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12712024-07-26T02:35:06.099ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12722024-07-26T02:35:06.099ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12732024-07-26T02:35:06.118ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12742024-07-26T02:35:06.118ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12752024-07-26T02:35:06.136ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
12762024-07-26T02:35:06.136ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
12772024-07-26T02:35:06.136ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
12782024-07-26T02:35:06.136ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
12792024-07-26T02:35:06.168ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
12802024-07-26T02:35:06.168ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
12812024-07-26T02:35:06.168ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
12822024-07-26T02:35:06.168ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
12832024-07-26T02:35:06.168ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
12842024-07-26T02:35:06.168ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3", TimeNotSynchronized), ("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8", TimeNotSynchronized), ("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63", TimeNotSynchronized), ("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c", TimeNotSynchronized), ("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 187888 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52946 req_id = fba67079-fe87-4066-bc30-356b3c7df1c5 response_code = 500 uri = /omicron-zones
12852024-07-26T02:35:06.168ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "fba67079-fe87-4066-bc30-356b3c7df1c5", "content-length": "124", "date": "Fri, 26 Jul 2024 02:35:05 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "fba67079-fe87-4066-bc30-356b3c7df1c5" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 6.823457524s sled_address = [fd00:1122:3344:101::1]:12345
12862024-07-26T02:35:12.975ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 25724e7e-f0f3-413d-ab41-b4a33973050b, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 82379658-84e2-487b-bcd4-b30ffc1d9874, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0eb4388e-1e4e-49da-b41b-58746266c965, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bbb96e05-c4c2-4d74-93ab-32168e25b9fd, kind: Service { id: 0eb4388e-1e4e-49da-b41b-58746266c965 }, name: Name("ntp-0eb4388e-1e4e-49da-b41b-58746266c965"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 87])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } } } }, OmicronZoneConfig { id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
12872024-07-26T02:35:12.975ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
12882024-07-26T02:35:12.975ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12892024-07-26T02:35:12.995ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
12902024-07-26T02:35:12.995ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12912024-07-26T02:35:13.017ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
12922024-07-26T02:35:13.017ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12932024-07-26T02:35:13.035ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12942024-07-26T02:35:13.035ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12952024-07-26T02:35:13.055ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12962024-07-26T02:35:13.055ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12972024-07-26T02:35:13.074ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12982024-07-26T02:35:13.074ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12992024-07-26T02:35:13.093ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13002024-07-26T02:35:13.093ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13012024-07-26T02:35:13.111ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13022024-07-26T02:35:13.111ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13032024-07-26T02:35:13.128ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
13042024-07-26T02:35:13.129ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
13052024-07-26T02:35:13.129ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
13062024-07-26T02:35:13.129ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
13072024-07-26T02:35:13.161ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
13082024-07-26T02:35:13.161ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
13092024-07-26T02:35:13.161ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
13102024-07-26T02:35:13.161ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
13112024-07-26T02:35:13.161ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
13122024-07-26T02:35:13.161ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8", TimeNotSynchronized), ("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6", TimeNotSynchronized), ("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c", TimeNotSynchronized), ("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63", TimeNotSynchronized), ("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 186905 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52946 req_id = 8a83e676-25a2-4f88-9555-6f718836b2ab response_code = 500 uri = /omicron-zones
13132024-07-26T02:35:13.161ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "8a83e676-25a2-4f88-9555-6f718836b2ab", "content-length": "124", "date": "Fri, 26 Jul 2024 02:35:12 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "8a83e676-25a2-4f88-9555-6f718836b2ab" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 5.165478931s sled_address = [fd00:1122:3344:101::1]:12345
13142024-07-26T02:35:16.345ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:41568
13152024-07-26T02:35:16.345ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 81 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:41568 req_id = 24a7a00c-f367-4367-a1a8-18563e65e77b response_code = 200 uri = /baseboard
13162024-07-26T02:35:18.315ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 25724e7e-f0f3-413d-ab41-b4a33973050b, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 82379658-84e2-487b-bcd4-b30ffc1d9874, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0eb4388e-1e4e-49da-b41b-58746266c965, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bbb96e05-c4c2-4d74-93ab-32168e25b9fd, kind: Service { id: 0eb4388e-1e4e-49da-b41b-58746266c965 }, name: Name("ntp-0eb4388e-1e4e-49da-b41b-58746266c965"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 87])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } } } }, OmicronZoneConfig { id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
13172024-07-26T02:35:18.315ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
13182024-07-26T02:35:18.315ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13192024-07-26T02:35:18.336ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
13202024-07-26T02:35:18.336ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13212024-07-26T02:35:18.354ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
13222024-07-26T02:35:18.354ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13232024-07-26T02:35:18.373ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13242024-07-26T02:35:18.373ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13252024-07-26T02:35:18.391ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13262024-07-26T02:35:18.391ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13272024-07-26T02:35:18.409ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13282024-07-26T02:35:18.409ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13292024-07-26T02:35:18.426ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13302024-07-26T02:35:18.426ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13312024-07-26T02:35:18.444ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13322024-07-26T02:35:18.444ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13332024-07-26T02:35:18.461ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
13342024-07-26T02:35:18.461ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
13352024-07-26T02:35:18.461ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
13362024-07-26T02:35:18.461ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
13372024-07-26T02:35:18.492ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
13382024-07-26T02:35:18.492ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
13392024-07-26T02:35:18.492ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
13402024-07-26T02:35:18.492ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
13412024-07-26T02:35:18.492ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
13422024-07-26T02:35:18.492ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8", TimeNotSynchronized), ("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3", TimeNotSynchronized), ("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63", TimeNotSynchronized), ("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c", TimeNotSynchronized), ("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 177778 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52946 req_id = 62766a10-0ace-4f9f-a0d5-154e72e04005 response_code = 500 uri = /omicron-zones
13432024-07-26T02:35:18.492ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "62766a10-0ace-4f9f-a0d5-154e72e04005", "content-length": "124", "date": "Fri, 26 Jul 2024 02:35:18 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "62766a10-0ace-4f9f-a0d5-154e72e04005" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 4.652766869s sled_address = [fd00:1122:3344:101::1]:12345
13442024-07-26T02:35:21.865ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 2.361278625s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
13452024-07-26T02:35:23.133ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 25724e7e-f0f3-413d-ab41-b4a33973050b, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 82379658-84e2-487b-bcd4-b30ffc1d9874, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0eb4388e-1e4e-49da-b41b-58746266c965, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bbb96e05-c4c2-4d74-93ab-32168e25b9fd, kind: Service { id: 0eb4388e-1e4e-49da-b41b-58746266c965 }, name: Name("ntp-0eb4388e-1e4e-49da-b41b-58746266c965"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 87])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } } } }, OmicronZoneConfig { id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
13462024-07-26T02:35:23.133ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
13472024-07-26T02:35:23.133ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13482024-07-26T02:35:23.155ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
13492024-07-26T02:35:23.155ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13502024-07-26T02:35:23.173ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
13512024-07-26T02:35:23.173ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13522024-07-26T02:35:23.192ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13532024-07-26T02:35:23.193ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13542024-07-26T02:35:23.211ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13552024-07-26T02:35:23.211ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13562024-07-26T02:35:23.230ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13572024-07-26T02:35:23.230ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13582024-07-26T02:35:23.248ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13592024-07-26T02:35:23.248ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13602024-07-26T02:35:23.267ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13612024-07-26T02:35:23.267ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13622024-07-26T02:35:23.286ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
13632024-07-26T02:35:23.286ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
13642024-07-26T02:35:23.286ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
13652024-07-26T02:35:23.286ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
13662024-07-26T02:35:23.318ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
13672024-07-26T02:35:23.318ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
13682024-07-26T02:35:23.318ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
13692024-07-26T02:35:23.318ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
13702024-07-26T02:35:23.318ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
13712024-07-26T02:35:23.318ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8", TimeNotSynchronized), ("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3", TimeNotSynchronized), ("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63", TimeNotSynchronized), ("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c", TimeNotSynchronized), ("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 185977 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52946 req_id = 212506e8-041b-419e-a689-e560b2b82799 response_code = 500 uri = /omicron-zones
13722024-07-26T02:35:23.318ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "212506e8-041b-419e-a689-e560b2b82799", "content-length": "124", "date": "Fri, 26 Jul 2024 02:35:23 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "212506e8-041b-419e-a689-e560b2b82799" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 9.781757648s sled_address = [fd00:1122:3344:101::1]:12345
13732024-07-26T02:35:33.075ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 25724e7e-f0f3-413d-ab41-b4a33973050b, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 82379658-84e2-487b-bcd4-b30ffc1d9874, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0eb4388e-1e4e-49da-b41b-58746266c965, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bbb96e05-c4c2-4d74-93ab-32168e25b9fd, kind: Service { id: 0eb4388e-1e4e-49da-b41b-58746266c965 }, name: Name("ntp-0eb4388e-1e4e-49da-b41b-58746266c965"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 87])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } } } }, OmicronZoneConfig { id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
13742024-07-26T02:35:33.077ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
13752024-07-26T02:35:33.077ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13762024-07-26T02:35:33.098ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
13772024-07-26T02:35:33.098ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13782024-07-26T02:35:33.118ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
13792024-07-26T02:35:33.118ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13802024-07-26T02:35:33.137ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13812024-07-26T02:35:33.137ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13822024-07-26T02:35:33.156ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13832024-07-26T02:35:33.156ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13842024-07-26T02:35:33.174ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13852024-07-26T02:35:33.174ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13862024-07-26T02:35:33.192ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13872024-07-26T02:35:33.192ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13882024-07-26T02:35:33.211ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13892024-07-26T02:35:33.211ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13902024-07-26T02:35:33.229ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
13912024-07-26T02:35:33.229ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
13922024-07-26T02:35:33.229ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
13932024-07-26T02:35:33.229ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
13942024-07-26T02:35:33.262ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
13952024-07-26T02:35:33.262ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
13962024-07-26T02:35:33.262ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
13972024-07-26T02:35:33.262ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
13982024-07-26T02:35:33.262ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
13992024-07-26T02:35:33.262ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63", TimeNotSynchronized), ("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c", TimeNotSynchronized), ("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8", TimeNotSynchronized), ("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3", TimeNotSynchronized), ("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 187303 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52946 req_id = fcc37c55-72df-4dd7-9e5f-0e4c291a0aee response_code = 500 uri = /omicron-zones
14002024-07-26T02:35:33.263ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "fcc37c55-72df-4dd7-9e5f-0e4c291a0aee", "content-length": "124", "date": "Fri, 26 Jul 2024 02:35:33 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "fcc37c55-72df-4dd7-9e5f-0e4c291a0aee" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 10.700510248s sled_address = [fd00:1122:3344:101::1]:12345
14012024-07-26T02:35:39.218ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 4.94102194s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
14022024-07-26T02:35:43.936ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 25724e7e-f0f3-413d-ab41-b4a33973050b, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 82379658-84e2-487b-bcd4-b30ffc1d9874, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0eb4388e-1e4e-49da-b41b-58746266c965, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bbb96e05-c4c2-4d74-93ab-32168e25b9fd, kind: Service { id: 0eb4388e-1e4e-49da-b41b-58746266c965 }, name: Name("ntp-0eb4388e-1e4e-49da-b41b-58746266c965"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 87])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } } } }, OmicronZoneConfig { id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
14032024-07-26T02:35:43.936ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
14042024-07-26T02:35:43.936ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14052024-07-26T02:35:43.958ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
14062024-07-26T02:35:43.958ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14072024-07-26T02:35:43.977ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
14082024-07-26T02:35:43.977ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14092024-07-26T02:35:43.995ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
14102024-07-26T02:35:43.995ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14112024-07-26T02:35:44.013ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
14122024-07-26T02:35:44.013ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14132024-07-26T02:35:44.031ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
14142024-07-26T02:35:44.031ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14152024-07-26T02:35:44.050ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
14162024-07-26T02:35:44.050ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14172024-07-26T02:35:44.068ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
14182024-07-26T02:35:44.068ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14192024-07-26T02:35:44.088ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
14202024-07-26T02:35:44.088ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
14212024-07-26T02:35:44.088ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
14222024-07-26T02:35:44.088ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
14232024-07-26T02:35:44.121ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
14242024-07-26T02:35:44.121ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
14252024-07-26T02:35:44.121ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
14262024-07-26T02:35:44.121ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
14272024-07-26T02:35:44.121ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
14282024-07-26T02:35:44.121ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c", TimeNotSynchronized), ("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63", TimeNotSynchronized), ("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8", TimeNotSynchronized), ("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3", TimeNotSynchronized), ("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 185628 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52946 req_id = f0877265-abb8-4724-973b-3d5714310a6b response_code = 500 uri = /omicron-zones
14292024-07-26T02:35:44.121ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "f0877265-abb8-4724-973b-3d5714310a6b", "content-length": "124", "date": "Fri, 26 Jul 2024 02:35:43 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "f0877265-abb8-4724-973b-3d5714310a6b" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 10.618922343s sled_address = [fd00:1122:3344:101::1]:12345
14302024-07-26T02:35:46.296ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:39105
14312024-07-26T02:35:46.297ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 731 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:39105 req_id = 77d08283-c539-4772-9574-63bb9af07711 response_code = 200 uri = /baseboard
14322024-07-26T02:35:54.712ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 25724e7e-f0f3-413d-ab41-b4a33973050b, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 82379658-84e2-487b-bcd4-b30ffc1d9874, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0eb4388e-1e4e-49da-b41b-58746266c965, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bbb96e05-c4c2-4d74-93ab-32168e25b9fd, kind: Service { id: 0eb4388e-1e4e-49da-b41b-58746266c965 }, name: Name("ntp-0eb4388e-1e4e-49da-b41b-58746266c965"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 87])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } } } }, OmicronZoneConfig { id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
14332024-07-26T02:35:54.712ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
14342024-07-26T02:35:54.712ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14352024-07-26T02:35:54.733ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
14362024-07-26T02:35:54.733ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14372024-07-26T02:35:54.750ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
14382024-07-26T02:35:54.750ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14392024-07-26T02:35:54.768ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
14402024-07-26T02:35:54.768ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14412024-07-26T02:35:54.786ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
14422024-07-26T02:35:54.786ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14432024-07-26T02:35:54.804ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
14442024-07-26T02:35:54.804ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14452024-07-26T02:35:54.822ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
14462024-07-26T02:35:54.822ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14472024-07-26T02:35:54.840ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
14482024-07-26T02:35:54.840ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14492024-07-26T02:35:54.857ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
14502024-07-26T02:35:54.857ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
14512024-07-26T02:35:54.857ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
14522024-07-26T02:35:54.858ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
14532024-07-26T02:35:54.891ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
14542024-07-26T02:35:54.891ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
14552024-07-26T02:35:54.891ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
14562024-07-26T02:35:54.891ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
14572024-07-26T02:35:54.891ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
14582024-07-26T02:35:54.891ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8", TimeNotSynchronized), ("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3", TimeNotSynchronized), ("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c", TimeNotSynchronized), ("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63", TimeNotSynchronized), ("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 179674 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52946 req_id = 615f534f-4017-4f68-b565-479d0c27c7d2 response_code = 500 uri = /omicron-zones
14592024-07-26T02:35:54.891ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "615f534f-4017-4f68-b565-479d0c27c7d2", "content-length": "124", "date": "Fri, 26 Jul 2024 02:35:54 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "615f534f-4017-4f68-b565-479d0c27c7d2" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 21.177105438s sled_address = [fd00:1122:3344:101::1]:12345
14602024-07-26T02:35:59.143ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 7.360949378s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
14612024-07-26T02:36:16.012ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 25724e7e-f0f3-413d-ab41-b4a33973050b, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 82379658-84e2-487b-bcd4-b30ffc1d9874, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0eb4388e-1e4e-49da-b41b-58746266c965, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bbb96e05-c4c2-4d74-93ab-32168e25b9fd, kind: Service { id: 0eb4388e-1e4e-49da-b41b-58746266c965 }, name: Name("ntp-0eb4388e-1e4e-49da-b41b-58746266c965"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 87])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } } } }, OmicronZoneConfig { id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
14622024-07-26T02:36:16.012ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
14632024-07-26T02:36:16.012ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14642024-07-26T02:36:16.034ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
14652024-07-26T02:36:16.034ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14662024-07-26T02:36:16.054ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
14672024-07-26T02:36:16.054ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14682024-07-26T02:36:16.072ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
14692024-07-26T02:36:16.072ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14702024-07-26T02:36:16.091ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
14712024-07-26T02:36:16.091ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14722024-07-26T02:36:16.109ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
14732024-07-26T02:36:16.109ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14742024-07-26T02:36:16.126ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
14752024-07-26T02:36:16.126ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14762024-07-26T02:36:16.145ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
14772024-07-26T02:36:16.145ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14782024-07-26T02:36:16.163ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
14792024-07-26T02:36:16.163ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
14802024-07-26T02:36:16.163ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
14812024-07-26T02:36:16.164ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
14822024-07-26T02:36:16.195ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
14832024-07-26T02:36:16.195ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
14842024-07-26T02:36:16.195ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
14852024-07-26T02:36:16.195ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
14862024-07-26T02:36:16.195ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
14872024-07-26T02:36:16.195ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8", TimeNotSynchronized), ("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6", TimeNotSynchronized), ("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63", TimeNotSynchronized), ("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c", TimeNotSynchronized), ("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 183752 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52946 req_id = 091d79f8-ecd6-41f6-95c6-ef63081634e2 response_code = 500 uri = /omicron-zones
14882024-07-26T02:36:16.195ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "091d79f8-ecd6-41f6-95c6-ef63081634e2", "content-length": "124", "date": "Fri, 26 Jul 2024 02:36:16 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "091d79f8-ecd6-41f6-95c6-ef63081634e2" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 21.724523078s sled_address = [fd00:1122:3344:101::1]:12345
14892024-07-26T02:36:16.250ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:45475
14902024-07-26T02:36:16.250ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 53 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:45475 req_id = 4df9d096-350b-4b0e-9400-cadd8f29d6df response_code = 200 uri = /baseboard
14912024-07-26T02:36:21.478ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 23.70618451s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
14922024-07-26T02:36:37.862ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 25724e7e-f0f3-413d-ab41-b4a33973050b, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 82379658-84e2-487b-bcd4-b30ffc1d9874, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0eb4388e-1e4e-49da-b41b-58746266c965, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bbb96e05-c4c2-4d74-93ab-32168e25b9fd, kind: Service { id: 0eb4388e-1e4e-49da-b41b-58746266c965 }, name: Name("ntp-0eb4388e-1e4e-49da-b41b-58746266c965"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 87])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } } } }, OmicronZoneConfig { id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
14932024-07-26T02:36:37.862ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
14942024-07-26T02:36:37.862ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14952024-07-26T02:36:37.885ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
14962024-07-26T02:36:37.885ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14972024-07-26T02:36:37.902ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
14982024-07-26T02:36:37.902ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14992024-07-26T02:36:37.920ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
15002024-07-26T02:36:37.920ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15012024-07-26T02:36:37.937ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
15022024-07-26T02:36:37.937ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15032024-07-26T02:36:37.955ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
15042024-07-26T02:36:37.956ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15052024-07-26T02:36:37.972ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
15062024-07-26T02:36:37.972ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15072024-07-26T02:36:37.989ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
15082024-07-26T02:36:37.989ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15092024-07-26T02:36:38.008ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
15102024-07-26T02:36:38.008ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
15112024-07-26T02:36:38.008ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
15122024-07-26T02:36:38.008ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
15132024-07-26T02:36:38.040ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
15142024-07-26T02:36:38.040ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
15152024-07-26T02:36:38.040ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
15162024-07-26T02:36:38.040ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
15172024-07-26T02:36:38.040ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3062 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
15182024-07-26T02:36:38.040ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c", TimeNotSynchronized), ("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8", TimeNotSynchronized), ("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63", TimeNotSynchronized), ("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6", TimeNotSynchronized), ("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:902 latency_us = 178453 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52946 req_id = bbd81dce-6d71-4d91-9ab4-76778af06e2b response_code = 500 uri = /omicron-zones
15192024-07-26T02:36:38.040ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "bbd81dce-6d71-4d91-9ab4-76778af06e2b", "content-length": "124", "date": "Fri, 26 Jul 2024 02:36:38 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "bbd81dce-6d71-4d91-9ab4-76778af06e2b" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 34.873448755s sled_address = [fd00:1122:3344:101::1]:12345
15202024-07-26T02:36:46.202ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:49178
15212024-07-26T02:36:46.202ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 34 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:49178 req_id = c508f4dc-39ee-4b67-8f86-ea7354634e48 response_code = 200 uri = /baseboard
15222024-07-26T02:37:00.115ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 16.3372196s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
15232024-07-26T02:37:12.856ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 25724e7e-f0f3-413d-ab41-b4a33973050b, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 82379658-84e2-487b-bcd4-b30ffc1d9874, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0eb4388e-1e4e-49da-b41b-58746266c965, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bbb96e05-c4c2-4d74-93ab-32168e25b9fd, kind: Service { id: 0eb4388e-1e4e-49da-b41b-58746266c965 }, name: Name("ntp-0eb4388e-1e4e-49da-b41b-58746266c965"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 87])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } } } }, OmicronZoneConfig { id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
15242024-07-26T02:37:12.857ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
15252024-07-26T02:37:12.857ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15262024-07-26T02:37:12.879ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
15272024-07-26T02:37:12.879ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15282024-07-26T02:37:12.897ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
15292024-07-26T02:37:12.897ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15302024-07-26T02:37:12.914ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
15312024-07-26T02:37:12.914ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15322024-07-26T02:37:12.932ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
15332024-07-26T02:37:12.932ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15342024-07-26T02:37:12.951ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
15352024-07-26T02:37:12.951ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15362024-07-26T02:37:12.969ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
15372024-07-26T02:37:12.969ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15382024-07-26T02:37:12.987ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
15392024-07-26T02:37:12.987ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15402024-07-26T02:37:13.006ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
15412024-07-26T02:37:13.006ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
15422024-07-26T02:37:13.006ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
15432024-07-26T02:37:13.006ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
15442024-07-26T02:37:13.089ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
15452024-07-26T02:37:13.089ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
15462024-07-26T02:37:13.089ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
15472024-07-26T02:37:13.089ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
15482024-07-26T02:37:13.089ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
15492024-07-26T02:37:13.174ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6 file = illumos-utils/src/zone.rs:316
15502024-07-26T02:37:13.178ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3 file = illumos-utils/src/zone.rs:316
15512024-07-26T02:37:13.183ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8 file = illumos-utils/src/zone.rs:316
15522024-07-26T02:37:13.188ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c file = illumos-utils/src/zone.rs:316
15532024-07-26T02:37:13.192ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63 file = illumos-utils/src/zone.rs:316
15542024-07-26T02:37:13.197ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6 file = illumos-utils/src/zone.rs:354
15552024-07-26T02:37:13.202ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3 file = illumos-utils/src/zone.rs:354
15562024-07-26T02:37:13.206ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8 file = illumos-utils/src/zone.rs:354
15572024-07-26T02:37:13.211ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c file = illumos-utils/src/zone.rs:354
15582024-07-26T02:37:13.215ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63 file = illumos-utils/src/zone.rs:354
15592024-07-26T02:37:16.073ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService5'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::6]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
15602024-07-26T02:37:16.073ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
15612024-07-26T02:37:16.122ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService9'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='5f42bb99-520b-4711-ae44-83f79f9b2f63'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::7]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
15622024-07-26T02:37:16.123ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
15632024-07-26T02:37:16.183ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService8'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::4'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='3f64979c-8afb-403e-8824-2c7f0af65e0c'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::4]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
15642024-07-26T02:37:16.183ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
15652024-07-26T02:37:16.200ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:34438
15662024-07-26T02:37:16.200ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 63 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:34438 req_id = fc4af6ff-fe59-45bd-a386-2a1657404ed7 response_code = 200 uri = /baseboard
15672024-07-26T02:37:16.226ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6") failed: Property not found. retry in 28.348864ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
15682024-07-26T02:37:16.233ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService6'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='dd312e93-ad11-47cc-8ead-ef2eaf27f3a3'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::5]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
15692024-07-26T02:37:16.233ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
15702024-07-26T02:37:16.253ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService7'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::3'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='31365180-26ce-40d1-a6d6-dd6b5fa83fe8'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::3]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
15712024-07-26T02:37:16.253ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
15722024-07-26T02:37:16.266ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63") failed: Property not found. retry in 25.680438ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
15732024-07-26T02:37:16.275ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6") failed: Property not found. retry in 124.765604ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
15742024-07-26T02:37:16.319ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63") failed: Property not found. retry in 86.072486ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
15752024-07-26T02:37:16.490ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c") failed: Property not found. retry in 41.289359ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
15762024-07-26T02:37:16.802ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6") failed: Property not found. retry in 269.724923ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
15772024-07-26T02:37:16.919ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63") failed: Property not found. retry in 122.816043ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
15782024-07-26T02:37:17.037ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c") failed: Property not found. retry in 131.541212ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
15792024-07-26T02:37:17.179ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63") failed: Property not found. retry in 508.204618ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
15802024-07-26T02:37:17.325ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6") failed: Property not found. retry in 308.71171ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
15812024-07-26T02:37:17.386ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3") failed: Property not found. retry in 48.130816ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
15822024-07-26T02:37:17.446ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c") failed: Property not found. retry in 203.586688ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
15832024-07-26T02:37:17.557ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3") failed: Property not found. retry in 51.187971ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
15842024-07-26T02:37:17.627ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8") failed: Property not found. retry in 31.501421ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
15852024-07-26T02:37:17.723ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6") failed: Property not found. retry in 560.721637ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
15862024-07-26T02:37:17.782ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3") failed: Property not found. retry in 185.490796ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
15872024-07-26T02:37:17.814ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8") failed: Property not found. retry in 81.840943ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
15882024-07-26T02:37:17.846ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c") failed: Property not found. retry in 326.194868ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
15892024-07-26T02:37:17.870ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63") failed: Property not found. retry in 818.913891ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
15902024-07-26T02:37:17.916ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8") failed: Property not found. retry in 298.190969ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
15912024-07-26T02:37:17.987ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3") failed: Property not found. retry in 338.21753ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
15922024-07-26T02:37:18.194ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c") failed: Property not found. retry in 1.168605502s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
15932024-07-26T02:37:18.238ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8") failed: Property not found. retry in 582.526718ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
15942024-07-26T02:37:18.302ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6") failed: Property not found. retry in 823.491703ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
15952024-07-26T02:37:18.347ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3") failed: Property not found. retry in 1.061350432s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
15962024-07-26T02:37:18.885ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63") failed: Property not found. retry in 1.250754661s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
15972024-07-26T02:37:18.897ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8") failed: Property not found. retry in 683.207033ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
15982024-07-26T02:37:19.140ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6") failed: Property not found. retry in 1.403752628s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
15992024-07-26T02:37:19.374ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c") failed: Property not found. retry in 608.325449ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
16002024-07-26T02:37:19.418ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3") failed: Property not found. retry in 721.549449ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
16012024-07-26T02:37:19.590ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8") failed: Property not found. retry in 782.30112ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
16022024-07-26T02:37:19.992ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c") failed: Property not found. retry in 520.598787ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
16032024-07-26T02:37:20.146ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63") failed: Property not found. retry in 1.133632244s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
16042024-07-26T02:37:20.153ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3") failed: Property not found. retry in 1.034531349s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
16052024-07-26T02:37:20.382ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8") failed: Property not found. retry in 1.143359559s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
16062024-07-26T02:37:20.523ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c") failed: Property not found. retry in 708.434097ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
16072024-07-26T02:37:20.553ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6") failed: Property not found. retry in 1.190795902s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
16082024-07-26T02:37:21.197ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3") failed: Property not found. retry in 1.233366511s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
16092024-07-26T02:37:21.240ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c") failed: Property not found. retry in 764.993467ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
16102024-07-26T02:37:21.288ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63") failed: Property not found. retry in 887.964392ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
16112024-07-26T02:37:21.534ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8") failed: Property not found. retry in 938.388542ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
16122024-07-26T02:37:21.753ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6") failed: Property not found. retry in 1.168093895s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
16132024-07-26T02:37:22.020ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c") failed: Property not found. retry in 974.078767ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
16142024-07-26T02:37:22.187ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63") failed: Property not found. retry in 595.894622ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
16152024-07-26T02:37:22.441ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3") failed: Property not found. retry in 797.833994ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
16162024-07-26T02:37:22.481ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8") failed: Property not found. retry in 788.85729ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
16172024-07-26T02:37:22.793ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63") failed: Property not found. retry in 1.384337995s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
16182024-07-26T02:37:22.932ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6") failed: Property not found. retry in 820.49341ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
16192024-07-26T02:37:23.005ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c") failed: Property not found. retry in 907.690129ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
16202024-07-26T02:37:23.248ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3") failed: Property not found. retry in 1.495564649s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
16212024-07-26T02:37:23.281ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8") failed: Property not found. retry in 1.081145555s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
16222024-07-26T02:37:23.761ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6") failed: Property not found. retry in 566.086741ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
16232024-07-26T02:37:23.922ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c") failed: Property not found. retry in 929.492741ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
16242024-07-26T02:37:24.187ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63") failed: Property not found. retry in 1.401143149s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
16252024-07-26T02:37:24.336ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6") failed: Property not found. retry in 725.650675ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
16262024-07-26T02:37:24.371ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8") failed: Property not found. retry in 942.726238ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
16272024-07-26T02:37:24.753ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3") failed: Property not found. retry in 1.473989812s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
16282024-07-26T02:37:24.861ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c") failed: Property not found. retry in 1.453465682s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
16292024-07-26T02:37:25.072ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6") failed: Property not found. retry in 1.055013492s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
16302024-07-26T02:37:25.324ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8") failed: Property not found. retry in 994.79164ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
16312024-07-26T02:37:25.598ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63") failed: Property not found. retry in 1.106531857s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
16322024-07-26T02:37:26.138ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6") failed: Property not found. retry in 882.509841ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
16332024-07-26T02:37:26.236ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3") failed: Property not found. retry in 766.546406ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
16342024-07-26T02:37:26.323ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c") failed: Property not found. retry in 1.343264458s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
16352024-07-26T02:37:26.331ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8") failed: Property not found. retry in 591.839421ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
16362024-07-26T02:37:26.714ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63") failed: Property not found. retry in 1.035664564s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
16372024-07-26T02:37:26.933ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8") failed: Property not found. retry in 883.763775ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
16382024-07-26T02:37:27.013ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3") failed: Property not found. retry in 1.132701167s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
16392024-07-26T02:37:27.029ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6") failed: Property not found. retry in 597.642464ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
16402024-07-26T02:37:27.637ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6") failed: Property not found. retry in 509.598542ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
16412024-07-26T02:37:28.168ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6") failed: Property not found. retry in 802.874275ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
16422024-07-26T02:37:28.981ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6") failed: Property not found. retry in 1.219860824s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
16432024-07-26T02:37:30.214ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6") failed: Property not found. retry in 1.374930606s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
16442024-07-26T02:37:31.600ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6") failed: Property not found. retry in 576.195772ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
16452024-07-26T02:37:32.187ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6") failed: Property not found. retry in 1.255048057s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
16462024-07-26T02:37:32.550ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 55.643777239s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
16472024-07-26T02:37:33.456ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6") failed: Property not found. retry in 1.486172572s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
16482024-07-26T02:37:34.953ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6") failed: Property not found. retry in 801.922264ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
16492024-07-26T02:37:35.766ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6") failed: Property not found. retry in 1.221089642s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
16502024-07-26T02:37:36.998ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6") failed: Property not found. retry in 1.038808169s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
16512024-07-26T02:37:38.048ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6") failed: Property not found. retry in 877.948863ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
16522024-07-26T02:37:38.936ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6") failed: Property not found. retry in 660.291266ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
16532024-07-26T02:37:39.606ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6") failed: Property not found. retry in 723.022259ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
16542024-07-26T02:37:40.338ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6") failed: Property not found. retry in 1.086410964s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
16552024-07-26T02:37:41.434ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6") failed: Property not found. retry in 1.202212782s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
16562024-07-26T02:37:42.646ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6") failed: Property not found. retry in 858.971062ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
16572024-07-26T02:37:43.514ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6") failed: Property not found. retry in 1.190622235s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
16582024-07-26T02:37:44.717ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6") failed: Property not found. retry in 1.488953671s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
16592024-07-26T02:37:46.228ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3058 zone = oxz_cockroachdb_9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6
16602024-07-26T02:37:46.228ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3058 zone = oxz_cockroachdb_dd312e93-ad11-47cc-8ead-ef2eaf27f3a3
16612024-07-26T02:37:46.228ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3058 zone = oxz_cockroachdb_31365180-26ce-40d1-a6d6-dd6b5fa83fe8
16622024-07-26T02:37:46.228ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3058 zone = oxz_cockroachdb_3f64979c-8afb-403e-8824-2c7f0af65e0c
16632024-07-26T02:37:46.228ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3058 zone = oxz_cockroachdb_5f42bb99-520b-4711-ae44-83f79f9b2f63
16642024-07-26T02:37:46.228ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
16652024-07-26T02:37:46.234ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:37805
16662024-07-26T02:37:46.235ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 238 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:37805 req_id = f4551aa7-6ddd-4cde-b253-4dd3db5a6481 response_code = 200 uri = /baseboard
16672024-07-26T02:37:46.274ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
16682024-07-26T02:37:46.374ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 33516914 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52946 req_id = a97c8d42-c790-4e03-be72-5aeb0a466fbd response_code = 204 uri = /omicron-zones
16692024-07-26T02:37:46.413ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:51705
16702024-07-26T02:37:46.441ZINFOSledAgent (ServiceManager): Initializing CRDB Cluster - sending request to [fd00:1122:3344:101::3]:32221 file = sled-agent/src/services.rs:3481
16712024-07-26T02:37:50.306ZINFOSledAgent (ServiceManager): Formatting CRDB file = sled-agent/src/services.rs:3499
16722024-07-26T02:37:59.631ZINFOSledAgent (ServiceManager): Formatting CRDB - Completed file = sled-agent/src/services.rs:3522
16732024-07-26T02:37:59.631ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 13218177 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:51705 req_id = 592045c1-22c3-450d-900e-9423a2ef8cb9 response_code = 204 uri = /cockroachdb
16742024-07-26T02:37:59.666ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 25724e7e-f0f3-413d-ab41-b4a33973050b, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 82379658-84e2-487b-bcd4-b30ffc1d9874, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0eb4388e-1e4e-49da-b41b-58746266c965, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: bbb96e05-c4c2-4d74-93ab-32168e25b9fd, kind: Service { id: 0eb4388e-1e4e-49da-b41b-58746266c965 }, name: Name("ntp-0eb4388e-1e4e-49da-b41b-58746266c965"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 87])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } } } }, OmicronZoneConfig { id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External } } } }, OmicronZoneConfig { id: cf7587f8-33d7-4e89-bad7-ae8cdd556425, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: 2f46bf86-4fc6-46a8-b573-bf2fb626f25d, kind: Service { id: cf7587f8-33d7-4e89-bad7-ae8cdd556425 }, name: Name("external-dns-cf7587f8-33d7-4e89-bad7-ae8cdd556425"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 168, 111])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 53979846-b429-4bae-9968-d96ff860d07e, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 567092fa-fb08-4940-8635-631e5c908557, kind: Service { id: 53979846-b429-4bae-9968-d96ff860d07e }, name: Name("external-dns-53979846-b429-4bae-9968-d96ff860d07e"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 158, 230])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 8b4031b8-df71-45f8-b977-b6aa00214087, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 136ed548-ab63-44da-a187-0fa13dc111a0 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 824ae30e-19ba-423f-923a-3afa2a59b03f, kind: Service { id: 8b4031b8-df71-45f8-b977-b6aa00214087 }, name: Name("nexus-8b4031b8-df71-45f8-b977-b6aa00214087"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 145])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 56f1d4b9-214d-4104-bc80-a770b9be089e, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: c5566b16-9a9f-498b-a4b1-83ac43308db5 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: e94e36b7-3a4e-4066-8a04-518f25a7ffdb, kind: Service { id: 56f1d4b9-214d-4104-bc80-a770b9be089e }, name: Name("nexus-56f1d4b9-214d-4104-bc80-a770b9be089e"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 180, 23])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 9d80bc64-2ec6-4d4e-96d7-5c012d075087, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: cb102b97-0352-4c03-8d37-4c80c4475510 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 95c06611-4118-4e93-84ac-1b68f8ff2a6d, kind: Service { id: 9d80bc64-2ec6-4d4e-96d7-5c012d075087 }, name: Name("nexus-9d80bc64-2ec6-4d4e-96d7-5c012d075087"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 148])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8752c2d7-36d8-49ed-860d-ec196a32b72c, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: d6874d80-d20e-408e-91a2-c5a467770626, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 78641540-4174-48c6-892e-29962a5829c6, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 2cc93cb4-8546-4d34-af83-babf2273221c, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: cb102b97-0352-4c03-8d37-4c80c4475510 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 290da25c-6635-4875-bb74-2c418e3e33a5, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: cb102b97-0352-4c03-8d37-4c80c4475510 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 0a17296b-a160-4ef4-9cb7-01d58322092f, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External } } } }, OmicronZoneConfig { id: 2fa8c446-5b0d-456d-a750-28a177c9ab27, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External } } } }, OmicronZoneConfig { id: b56f7823-5e5f-4958-a8ee-f08cb6a361c1, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External } } } }, OmicronZoneConfig { id: 07eec7c5-c6b0-4973-a922-2fdc5f972621, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External } } } }, OmicronZoneConfig { id: 39533c6a-2c55-4972-8616-1a19033efc8c, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: cb102b97-0352-4c03-8d37-4c80c4475510 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cb102b97-0352-4c03-8d37-4c80c4475510 (zpool), kind: External } } } }, OmicronZoneConfig { id: 2fc14079-6244-4be0-84a1-1c701868ee2d, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: c5566b16-9a9f-498b-a4b1-83ac43308db5 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c5566b16-9a9f-498b-a4b1-83ac43308db5 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7832441b-d9a6-4b73-a71f-8fb8e621ab9f, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 136ed548-ab63-44da-a187-0fa13dc111a0 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 136ed548-ab63-44da-a187-0fa13dc111a0 (zpool), kind: External } } } }, OmicronZoneConfig { id: 636d501d-962a-441b-8898-3876ad1cb8b3, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 3ed9ecd9-4aeb-4ec2-8876-b9c8ed7df016 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3ed9ecd9-4aeb-4ec2-8876-b9c8ed7df016 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
16752024-07-26T02:37:59.666ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:62928
16762024-07-26T02:37:59.667ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
16772024-07-26T02:37:59.667ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 25724e7e-f0f3-413d-ab41-b4a33973050b, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: InternalDns }, responder: ... }
16782024-07-26T02:37:59.690ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
16792024-07-26T02:37:59.690ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 4e64b0cb-e3e2-4959-adf4-d5cd70f25b40, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: InternalDns }, responder: ... }
16802024-07-26T02:37:59.713ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
16812024-07-26T02:37:59.713ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 82379658-84e2-487b-bcd4-b30ffc1d9874, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: InternalDns }, responder: ... }
16822024-07-26T02:37:59.734ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
16832024-07-26T02:37:59.734ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 31365180-26ce-40d1-a6d6-dd6b5fa83fe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16842024-07-26T02:37:59.753ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
16852024-07-26T02:37:59.753ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3f64979c-8afb-403e-8824-2c7f0af65e0c, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16862024-07-26T02:37:59.774ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
16872024-07-26T02:37:59.774ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: dd312e93-ad11-47cc-8ead-ef2eaf27f3a3, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16882024-07-26T02:37:59.794ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
16892024-07-26T02:37:59.794ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 9374a3cd-ae44-4f04-9fc8-6112fb6fc7f6, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16902024-07-26T02:37:59.814ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
16912024-07-26T02:37:59.814ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5f42bb99-520b-4711-ae44-83f79f9b2f63, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16922024-07-26T02:37:59.834ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cf7587f8-33d7-4e89-bad7-ae8cdd556425, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
16932024-07-26T02:37:59.834ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: cf7587f8-33d7-4e89-bad7-ae8cdd556425, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
16942024-07-26T02:37:59.909ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 53979846-b429-4bae-9968-d96ff860d07e, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
16952024-07-26T02:37:59.909ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 53979846-b429-4bae-9968-d96ff860d07e, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
16962024-07-26T02:37:59.985ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d6874d80-d20e-408e-91a2-c5a467770626, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:405
16972024-07-26T02:37:59.985ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: d6874d80-d20e-408e-91a2-c5a467770626, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
16982024-07-26T02:38:00.067ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0a17296b-a160-4ef4-9cb7-01d58322092f, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
16992024-07-26T02:38:00.067ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0a17296b-a160-4ef4-9cb7-01d58322092f, dataset_name: DatasetName { pool_name: ZpoolName { id: 4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8 (zpool), kind: External }, kind: Crucible }, responder: ... }
17002024-07-26T02:38:00.146ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2fa8c446-5b0d-456d-a750-28a177c9ab27, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
17012024-07-26T02:38:00.146ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 2fa8c446-5b0d-456d-a750-28a177c9ab27, dataset_name: DatasetName { pool_name: ZpoolName { id: b8ef9471-5b81-40ed-90b5-1103a4a9eb91 (zpool), kind: External }, kind: Crucible }, responder: ... }
17022024-07-26T02:38:00.222ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
17032024-07-26T02:38:00.222ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23, dataset_name: DatasetName { pool_name: ZpoolName { id: a47af5d0-e565-4513-84ad-c553b7265a72 (zpool), kind: External }, kind: Crucible }, responder: ... }
17042024-07-26T02:38:00.290ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b56f7823-5e5f-4958-a8ee-f08cb6a361c1, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
17052024-07-26T02:38:00.290ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: b56f7823-5e5f-4958-a8ee-f08cb6a361c1, dataset_name: DatasetName { pool_name: ZpoolName { id: 2a33f7b1-2bfd-4239-9138-fb534f4ccb3f (zpool), kind: External }, kind: Crucible }, responder: ... }
17062024-07-26T02:38:00.376ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 07eec7c5-c6b0-4973-a922-2fdc5f972621, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
17072024-07-26T02:38:00.377ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 07eec7c5-c6b0-4973-a922-2fdc5f972621, dataset_name: DatasetName { pool_name: ZpoolName { id: a38a1ea7-44a1-411f-bbb8-767c9a23135f (zpool), kind: External }, kind: Crucible }, responder: ... }
17082024-07-26T02:38:00.448ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 39533c6a-2c55-4972-8616-1a19033efc8c, dataset_name: DatasetName { pool_name: ZpoolName { id: cb102b97-0352-4c03-8d37-4c80c4475510 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
17092024-07-26T02:38:00.448ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 39533c6a-2c55-4972-8616-1a19033efc8c, dataset_name: DatasetName { pool_name: ZpoolName { id: cb102b97-0352-4c03-8d37-4c80c4475510 (zpool), kind: External }, kind: Crucible }, responder: ... }
17102024-07-26T02:38:00.518ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2fc14079-6244-4be0-84a1-1c701868ee2d, dataset_name: DatasetName { pool_name: ZpoolName { id: c5566b16-9a9f-498b-a4b1-83ac43308db5 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
17112024-07-26T02:38:00.518ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 2fc14079-6244-4be0-84a1-1c701868ee2d, dataset_name: DatasetName { pool_name: ZpoolName { id: c5566b16-9a9f-498b-a4b1-83ac43308db5 (zpool), kind: External }, kind: Crucible }, responder: ... }
17122024-07-26T02:38:00.583ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7832441b-d9a6-4b73-a71f-8fb8e621ab9f, dataset_name: DatasetName { pool_name: ZpoolName { id: 136ed548-ab63-44da-a187-0fa13dc111a0 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
17132024-07-26T02:38:00.583ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 7832441b-d9a6-4b73-a71f-8fb8e621ab9f, dataset_name: DatasetName { pool_name: ZpoolName { id: 136ed548-ab63-44da-a187-0fa13dc111a0 (zpool), kind: External }, kind: Crucible }, responder: ... }
17142024-07-26T02:38:00.649ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 636d501d-962a-441b-8898-3876ad1cb8b3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ed9ecd9-4aeb-4ec2-8876-b9c8ed7df016 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
17152024-07-26T02:38:00.649ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 636d501d-962a-441b-8898-3876ad1cb8b3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ed9ecd9-4aeb-4ec2-8876-b9c8ed7df016 (zpool), kind: External }, kind: Crucible }, responder: ... }
17162024-07-26T02:38:00.717ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
17172024-07-26T02:38:00.717ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
17182024-07-26T02:38:00.717ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
17192024-07-26T02:38:00.717ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
17202024-07-26T02:38:00.840ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:258
17212024-07-26T02:38:00.840ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
17222024-07-26T02:38:00.855ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:258
17232024-07-26T02:38:00.855ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
17242024-07-26T02:38:00.855ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
17252024-07-26T02:38:00.898ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
17262024-07-26T02:38:00.898ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
17272024-07-26T02:38:00.898ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
17282024-07-26T02:38:00.898ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
17292024-07-26T02:38:00.898ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
17302024-07-26T02:38:01.073ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:258
17312024-07-26T02:38:01.075ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:258
17322024-07-26T02:38:01.139ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:258
17332024-07-26T02:38:01.140ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
17342024-07-26T02:38:01.140ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
17352024-07-26T02:38:01.140ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
17362024-07-26T02:38:01.140ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
17372024-07-26T02:38:01.140ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
17382024-07-26T02:38:01.140ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
17392024-07-26T02:38:01.140ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621 file = illumos-utils/src/zone.rs:316
17402024-07-26T02:38:01.145ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c file = illumos-utils/src/zone.rs:316
17412024-07-26T02:38:01.150ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f file = illumos-utils/src/zone.rs:316
17422024-07-26T02:38:01.155ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23 file = illumos-utils/src/zone.rs:316
17432024-07-26T02:38:01.160ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27 file = illumos-utils/src/zone.rs:316
17442024-07-26T02:38:01.165ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626 file = illumos-utils/src/zone.rs:316
17452024-07-26T02:38:01.171ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5 file = illumos-utils/src/zone.rs:316
17462024-07-26T02:38:01.176ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c file = illumos-utils/src/zone.rs:316
17472024-07-26T02:38:01.301ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621 file = illumos-utils/src/zone.rs:354
17482024-07-26T02:38:01.306ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c file = illumos-utils/src/zone.rs:354
17492024-07-26T02:38:01.311ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f file = illumos-utils/src/zone.rs:354
17502024-07-26T02:38:01.316ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23 file = illumos-utils/src/zone.rs:354
17512024-07-26T02:38:01.321ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27 file = illumos-utils/src/zone.rs:354
17522024-07-26T02:38:01.326ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626 file = illumos-utils/src/zone.rs:354
17532024-07-26T02:38:01.331ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5 file = illumos-utils/src/zone.rs:354
17542024-07-26T02:38:01.336ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c file = illumos-utils/src/zone.rs:354
17552024-07-26T02:38:01.341ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f file = illumos-utils/src/zone.rs:316
17562024-07-26T02:38:01.347ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1 file = illumos-utils/src/zone.rs:316
17572024-07-26T02:38:01.354ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c file = illumos-utils/src/zone.rs:316
17582024-07-26T02:38:01.360ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d file = illumos-utils/src/zone.rs:316
17592024-07-26T02:38:01.366ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3 file = illumos-utils/src/zone.rs:316
17602024-07-26T02:38:01.372ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6 file = illumos-utils/src/zone.rs:316
17612024-07-26T02:38:01.380ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f file = illumos-utils/src/zone.rs:354
17622024-07-26T02:38:01.385ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1 file = illumos-utils/src/zone.rs:354
17632024-07-26T02:38:01.392ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c file = illumos-utils/src/zone.rs:354
17642024-07-26T02:38:01.398ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d file = illumos-utils/src/zone.rs:354
17652024-07-26T02:38:01.405ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3 file = illumos-utils/src/zone.rs:354
17662024-07-26T02:38:01.412ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6 file = illumos-utils/src/zone.rs:354
17672024-07-26T02:38:01.475ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:275
17682024-07-26T02:38:01.555ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:275
17692024-07-26T02:38:01.734ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:275
17702024-07-26T02:38:01.831ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:275
17712024-07-26T02:38:01.907ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:275
17722024-07-26T02:38:01.910ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:300
17732024-07-26T02:38:01.910ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:192
17742024-07-26T02:38:01.976ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:426 port = Port { inner: PortInner(PortData { name: "opte1", ip: 172.30.1.5, mac: MacAddr6([168, 64, 37, 255, 168, 111]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
17752024-07-26T02:38:01.976ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1207 zone_type = external_dns
17762024-07-26T02:38:01.979ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:300
17772024-07-26T02:38:01.980ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:192
17782024-07-26T02:38:02.039ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:426 port = Port { inner: PortInner(PortData { name: "opte2", ip: 172.30.2.6, mac: MacAddr6([168, 64, 37, 255, 180, 23]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
17792024-07-26T02:38:02.039ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1207 zone_type = nexus
17802024-07-26T02:38:02.043ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:300
17812024-07-26T02:38:02.043ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:192
17822024-07-26T02:38:02.103ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:426 port = Port { inner: PortInner(PortData { name: "opte3", ip: 172.30.1.6, mac: MacAddr6([168, 64, 37, 255, 158, 230]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
17832024-07-26T02:38:02.103ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1207 zone_type = external_dns
17842024-07-26T02:38:02.104ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:300
17852024-07-26T02:38:02.104ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:192
17862024-07-26T02:38:02.162ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:426 port = Port { inner: PortInner(PortData { name: "opte4", ip: 172.30.2.7, mac: MacAddr6([168, 64, 37, 255, 248, 148]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
17872024-07-26T02:38:02.162ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1207 zone_type = nexus
17882024-07-26T02:38:02.164ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:300
17892024-07-26T02:38:02.164ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:192
17902024-07-26T02:38:02.221ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:426 port = Port { inner: PortInner(PortData { name: "opte5", ip: 172.30.2.5, mac: MacAddr6([168, 64, 37, 255, 161, 145]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
17912024-07-26T02:38:02.222ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1207 zone_type = nexus
17922024-07-26T02:38:02.222ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.166 file = clients/dpd-client/src/lib.rs:83
17932024-07-26T02:38:02.222ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.166 file = clients/dpd-client/src/lib.rs:90
17942024-07-26T02:38:02.225ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.168 file = clients/dpd-client/src/lib.rs:83
17952024-07-26T02:38:02.225ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.168 file = clients/dpd-client/src/lib.rs:90
17962024-07-26T02:38:02.225ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.164 file = clients/dpd-client/src/lib.rs:83
17972024-07-26T02:38:02.225ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.164 file = clients/dpd-client/src/lib.rs:90
17982024-07-26T02:38:02.225ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.167 file = clients/dpd-client/src/lib.rs:83
17992024-07-26T02:38:02.225ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.167 file = clients/dpd-client/src/lib.rs:90
18002024-07-26T02:38:02.225ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.165 file = clients/dpd-client/src/lib.rs:83
18012024-07-26T02:38:02.225ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.165 file = clients/dpd-client/src/lib.rs:90
18022024-07-26T02:38:02.228ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.166 file = clients/dpd-client/src/lib.rs:118
18032024-07-26T02:38:02.230ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
18042024-07-26T02:38:02.230ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.168 file = clients/dpd-client/src/lib.rs:118
18052024-07-26T02:38:02.303ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
18062024-07-26T02:38:02.304ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.164 file = clients/dpd-client/src/lib.rs:118
18072024-07-26T02:38:02.308ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.167 file = clients/dpd-client/src/lib.rs:118
18082024-07-26T02:38:02.312ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.165 file = clients/dpd-client/src/lib.rs:118
18092024-07-26T02:38:02.312ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
18102024-07-26T02:38:02.312ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
18112024-07-26T02:38:02.573ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087 file = illumos-utils/src/zone.rs:316
18122024-07-26T02:38:02.713ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
18132024-07-26T02:38:03.277ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087 file = illumos-utils/src/zone.rs:316
18142024-07-26T02:38:03.284ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087 file = illumos-utils/src/zone.rs:354
18152024-07-26T02:38:03.294ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425 file = illumos-utils/src/zone.rs:316
18162024-07-26T02:38:03.303ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e file = illumos-utils/src/zone.rs:316
18172024-07-26T02:38:03.312ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e file = illumos-utils/src/zone.rs:316
18182024-07-26T02:38:03.321ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087 file = illumos-utils/src/zone.rs:354
18192024-07-26T02:38:03.331ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425 file = illumos-utils/src/zone.rs:354
18202024-07-26T02:38:03.346ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e file = illumos-utils/src/zone.rs:354
18212024-07-26T02:38:03.750ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e file = illumos-utils/src/zone.rs:354
18222024-07-26T02:38:08.295ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService11'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::10'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::10'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18232024-07-26T02:38:08.299ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c
18242024-07-26T02:38:08.357ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService16'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::11'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::11'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18252024-07-26T02:38:08.358ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService23'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::f'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::f'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18262024-07-26T02:38:08.359ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5
18272024-07-26T02:38:08.371ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6
18282024-07-26T02:38:08.415ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService13'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::14'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_a47af5d0-e565-4513-84ad-c553b7265a72/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::14'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='17585b17-0341-412d-9707-83bcf3544c21'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18292024-07-26T02:38:08.417ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService22'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_3ed9ecd9-4aeb-4ec2-8876-b9c8ed7df016/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1a'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='2e249ddd-60b3-4fb9-92cb-6874bafd60cc'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18302024-07-26T02:38:08.418ZINFOSledAgent (ServiceManager): Booting oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23
18312024-07-26T02:38:08.431ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService14'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::13'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_b8ef9471-5b81-40ed-90b5-1103a4a9eb91/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::13'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='c7756490-f3a2-4b02-b5e5-5a9799127a6b'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18322024-07-26T02:38:08.435ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService19'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::15'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_2a33f7b1-2bfd-4239-9138-fb534f4ccb3f/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::15'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='8632c9e6-3a2e-480f-a7e5-8b904bee9a8e'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18332024-07-26T02:38:08.435ZINFOSledAgent (ServiceManager): Booting oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3
18342024-07-26T02:38:08.450ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService10'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::16'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_a38a1ea7-44a1-411f-bbb8-767c9a23135f/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::16'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='88c84e03-d109-445a-859d-3c8577e7f34e'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18352024-07-26T02:38:08.454ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService12'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::19'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_136ed548-ab63-44da-a187-0fa13dc111a0/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::19'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='5a5d2529-0805-4e65-981f-8b40b25276c4'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18362024-07-26T02:38:08.454ZINFOSledAgent (ServiceManager): Booting oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27
18372024-07-26T02:38:08.463ZINFOSledAgent (ServiceManager): Booting oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1
18382024-07-26T02:38:08.474ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService20'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::17'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_cb102b97-0352-4c03-8d37-4c80c4475510/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::17'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='ef522bde-3d98-4c00-8a97-bedc4b2ab82d'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18392024-07-26T02:38:08.477ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService21'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::18'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_c5566b16-9a9f-498b-a4b1-83ac43308db5/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::18'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='d3737deb-9edf-4e48-87d6-5b99921a3cd9'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18402024-07-26T02:38:08.477ZINFOSledAgent (ServiceManager): Booting oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621
18412024-07-26T02:38:08.486ZINFOSledAgent (ServiceManager): Booting oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f
18422024-07-26T02:38:08.497ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService18'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::12'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::12'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='927bb11d-6f80-4b00-ae64-d69603228ff4'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18432024-07-26T02:38:08.497ZINFOSledAgent (ServiceManager): Booting oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c
18442024-07-26T02:38:08.507ZINFOSledAgent (ServiceManager): Booting oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d
18452024-07-26T02:38:08.516ZINFOSledAgent (ServiceManager): Booting oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f
18462024-07-26T02:38:08.757ZINFOSledAgent (ServiceManager): Profile for oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService17'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::d'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/oximeter"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="id" value='8752c2d7-36d8-49ed-860d-ec196a32b72c'/> <propval type="astring" name="address" value='[fd00:1122:3344:101::d]:12223'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18472024-07-26T02:38:08.757ZINFOSledAgent (ServiceManager): Booting oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c
18482024-07-26T02:38:09.227ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c") failed: Property not found. retry in 62.769952ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c
18492024-07-26T02:38:09.242ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6") failed: Property not found. retry in 38.064834ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6
18502024-07-26T02:38:09.256ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3") failed: Property not found. retry in 31.321397ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3
18512024-07-26T02:38:09.267ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f") failed: Property not found. retry in 60.245863ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f
18522024-07-26T02:38:09.278ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5") failed: Property not found. retry in 66.723357ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5
18532024-07-26T02:38:09.289ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621") failed: Property not found. retry in 60.038278ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621
18542024-07-26T02:38:09.300ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23") failed: Property not found. retry in 31.118792ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23
18552024-07-26T02:38:09.340ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27") failed: Property not found. retry in 63.032108ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27
18562024-07-26T02:38:09.389ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1") failed: Property not found. retry in 31.240491ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1
18572024-07-26T02:38:09.458ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c") failed: Property not found. retry in 73.670778ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c
18582024-07-26T02:38:09.500ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d") failed: Property not found. retry in 66.215613ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d
18592024-07-26T02:38:09.529ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3") failed: Property not found. retry in 134.263247ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3
18602024-07-26T02:38:09.555ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6") failed: Property not found. retry in 63.518001ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6
18612024-07-26T02:38:09.576ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621") failed: Property not found. retry in 132.828986ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621
18622024-07-26T02:38:09.595ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c") failed: Property not found. retry in 111.871589ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c
18632024-07-26T02:38:09.621ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f") failed: Property not found. retry in 68.121458ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f
18642024-07-26T02:38:09.763ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23") failed: Property not found. retry in 101.904253ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23
18652024-07-26T02:38:10.127ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27") failed: Property not found. retry in 139.762496ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27
18662024-07-26T02:38:10.199ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5") failed: Property not found. retry in 79.274963ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5
18672024-07-26T02:38:10.237ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c") failed: Property not found. retry in 42.369869ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c
18682024-07-26T02:38:10.368ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f") failed: Property not found. retry in 49.49862ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f
18692024-07-26T02:38:10.661ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1") failed: Property not found. retry in 142.712361ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1
18702024-07-26T02:38:10.707ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c") failed: Property not found. retry in 149.044898ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c
18712024-07-26T02:38:10.747ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d") failed: Property not found. retry in 70.931729ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d
18722024-07-26T02:38:10.764ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3") failed: Property not found. retry in 241.077888ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3
18732024-07-26T02:38:10.792ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6") failed: Property not found. retry in 133.927827ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6
18742024-07-26T02:38:10.819ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621") failed: Property not found. retry in 215.898263ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621
18752024-07-26T02:38:10.844ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c") failed: Property not found. retry in 142.510647ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c
18762024-07-26T02:38:10.864ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f") failed: Property not found. retry in 144.125509ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f
18772024-07-26T02:38:10.880ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23") failed: Property not found. retry in 233.852339ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23
18782024-07-26T02:38:10.895ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27") failed: Property not found. retry in 124.786925ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27
18792024-07-26T02:38:10.909ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5") failed: Property not found. retry in 250.587946ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5
18802024-07-26T02:38:10.922ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c") failed: Property not found. retry in 138.698342ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c
18812024-07-26T02:38:10.938ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f") failed: Property not found. retry in 114.572304ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f
18822024-07-26T02:38:10.951ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1") failed: Property not found. retry in 299.928998ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1
18832024-07-26T02:38:10.963ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c") failed: Property not found. retry in 154.987216ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c
18842024-07-26T02:38:10.978ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d") failed: Property not found. retry in 250.667616ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d
18852024-07-26T02:38:10.989ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6") failed: Property not found. retry in 251.141395ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6
18862024-07-26T02:38:11.001ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c") failed: Property not found. retry in 206.525984ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c
18872024-07-26T02:38:11.021ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3") failed: Property not found. retry in 270.652792ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3
18882024-07-26T02:38:11.034ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f") failed: Property not found. retry in 385.483751ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f
18892024-07-26T02:38:11.049ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27") failed: Property not found. retry in 372.7807ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27
18902024-07-26T02:38:11.063ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621") failed: Property not found. retry in 234.297889ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621
18912024-07-26T02:38:11.075ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c") failed: Property not found. retry in 235.876182ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c
18922024-07-26T02:38:11.088ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f") failed: Property not found. retry in 272.239825ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f
18932024-07-26T02:38:11.130ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23") failed: Property not found. retry in 492.033581ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23
18942024-07-26T02:38:11.145ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c") failed: Property not found. retry in 419.893031ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c
18952024-07-26T02:38:11.177ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5") failed: Property not found. retry in 333.90613ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5
18962024-07-26T02:38:11.221ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c") failed: Property not found. retry in 801.691172ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c
18972024-07-26T02:38:11.239ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d") failed: Property not found. retry in 393.881748ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d
18982024-07-26T02:38:11.250ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6") failed: Property not found. retry in 946.235077ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6
18992024-07-26T02:38:11.266ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1") failed: Property not found. retry in 360.712582ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1
19002024-07-26T02:38:11.305ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3") failed: Property not found. retry in 635.97115ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3
19012024-07-26T02:38:11.335ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621") failed: Property not found. retry in 438.452703ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621
19022024-07-26T02:38:11.347ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c") failed: Property not found. retry in 280.41401ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c
19032024-07-26T02:38:11.374ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f") failed: Property not found. retry in 400.079574ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f
19042024-07-26T02:38:11.435ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f") failed: Property not found. retry in 546.751043ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f
19052024-07-26T02:38:11.448ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27") failed: Property not found. retry in 539.115136ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27
19062024-07-26T02:38:11.524ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5") failed: Property not found. retry in 975.690754ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5
19072024-07-26T02:38:11.582ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c") failed: Property not found. retry in 726.845574ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c
19082024-07-26T02:38:11.637ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23") failed: Property not found. retry in 719.841747ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23
19092024-07-26T02:38:11.647ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c") failed: Property not found. retry in 925.243284ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c
19102024-07-26T02:38:11.661ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1") failed: Property not found. retry in 1.150291448s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1
19112024-07-26T02:38:11.678ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d") failed: Property not found. retry in 1.077458443s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d
19122024-07-26T02:38:11.726ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService28'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::9'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte3'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::9]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.6:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19132024-07-26T02:38:11.726ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e
19142024-07-26T02:38:11.789ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621") failed: Property not found. retry in 908.500712ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621
19152024-07-26T02:38:11.803ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f") failed: Property not found. retry in 502.955627ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f
19162024-07-26T02:38:11.959ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3") failed: Property not found. retry in 883.713327ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3
19172024-07-26T02:38:11.980ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e") failed: Property not found. retry in 61.438358ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e
19182024-07-26T02:38:11.994ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f") failed: Property not found. retry in 769.272173ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f
19192024-07-26T02:38:12.015ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27") failed: Property not found. retry in 528.668904ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27
19202024-07-26T02:38:12.043ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c") failed: Property not found. retry in 549.953662ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c
19212024-07-26T02:38:12.055ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e") failed: Property not found. retry in 76.026082ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e
19222024-07-26T02:38:12.143ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e") failed: Property not found. retry in 271.208066ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e
19232024-07-26T02:38:12.214ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6") failed: Property not found. retry in 1.299869702s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6
19242024-07-26T02:38:12.322ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f") failed: Property not found. retry in 757.378214ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f
19252024-07-26T02:38:12.337ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c") failed: Property not found. retry in 1.207117875s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c
19262024-07-26T02:38:12.376ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23") failed: Property not found. retry in 939.982446ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23
19272024-07-26T02:38:12.429ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e") failed: Property not found. retry in 565.363568ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e
19282024-07-26T02:38:12.513ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5") failed: Property not found. retry in 500.044382ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5
19292024-07-26T02:38:12.558ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27") failed: Property not found. retry in 1.051649901s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27
19302024-07-26T02:38:12.591ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c") failed: Property not found. retry in 998.529038ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c
19312024-07-26T02:38:12.607ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c") failed: Property not found. retry in 647.652897ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c
19322024-07-26T02:38:12.711ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621") failed: Property not found. retry in 1.175217273s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621
19332024-07-26T02:38:12.770ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d") failed: Property not found. retry in 1.347277943s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d
19342024-07-26T02:38:12.782ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f") failed: Property not found. retry in 540.030386ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f
19352024-07-26T02:38:12.825ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1") failed: Property not found. retry in 549.205695ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1
19362024-07-26T02:38:12.856ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3") failed: Property not found. retry in 1.051217282s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3
19372024-07-26T02:38:13.007ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e") failed: Property not found. retry in 591.971869ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e
19382024-07-26T02:38:13.025ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5") failed: Property not found. retry in 598.551122ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5
19392024-07-26T02:38:13.095ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f") failed: Property not found. retry in 694.714218ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f
19402024-07-26T02:38:13.266ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c") failed: Property not found. retry in 728.170445ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c
19412024-07-26T02:38:13.326ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23") failed: Property not found. retry in 531.813303ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23
19422024-07-26T02:38:13.336ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f") failed: Property not found. retry in 686.609232ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f
19432024-07-26T02:38:13.383ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1") failed: Property not found. retry in 1.212999918s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1
19442024-07-26T02:38:13.525ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6") failed: Property not found. retry in 1.053060177s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6
19452024-07-26T02:38:13.556ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c") failed: Property not found. retry in 638.215234ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c
19462024-07-26T02:38:13.601ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c") failed: Property not found. retry in 692.036866ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c
19472024-07-26T02:38:13.669ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e") failed: Property not found. retry in 1.157625353s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e
19482024-07-26T02:38:13.681ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27") failed: Property not found. retry in 703.566401ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27
19492024-07-26T02:38:13.692ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5") failed: Property not found. retry in 1.020846119s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5
19502024-07-26T02:38:13.804ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f") failed: Property not found. retry in 559.881043ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f
19512024-07-26T02:38:13.872ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23") failed: Property not found. retry in 759.721109ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23
19522024-07-26T02:38:13.898ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621") failed: Property not found. retry in 616.541054ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621
19532024-07-26T02:38:13.920ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3") failed: Property not found. retry in 1.089549693s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3
19542024-07-26T02:38:14.007ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c") failed: Property not found. retry in 1.418846584s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c
19552024-07-26T02:38:14.037ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f") failed: Property not found. retry in 658.991775ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f
19562024-07-26T02:38:14.130ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d") failed: Property not found. retry in 1.228968742s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d
19572024-07-26T02:38:14.259ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c") failed: Property not found. retry in 789.52309ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c
19582024-07-26T02:38:14.305ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c") failed: Property not found. retry in 679.592392ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c
19592024-07-26T02:38:14.377ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f") failed: Property not found. retry in 772.788959ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f
19602024-07-26T02:38:14.398ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27") failed: Property not found. retry in 1.252796076s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27
19612024-07-26T02:38:14.448ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService27'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte2'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19622024-07-26T02:38:14.448ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService25'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::c'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte4'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19632024-07-26T02:38:14.448ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService24'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte5'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19642024-07-26T02:38:14.448ZINFOSledAgent (ServiceManager): Booting oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e
19652024-07-26T02:38:14.455ZINFOSledAgent (ServiceManager): Booting oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087
19662024-07-26T02:38:14.461ZINFOSledAgent (ServiceManager): Booting oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087
19672024-07-26T02:38:14.529ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621") failed: Property not found. retry in 916.547566ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621
19682024-07-26T02:38:14.591ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6") failed: Property not found. retry in 1.223240924s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6
19692024-07-26T02:38:14.610ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1") failed: Property not found. retry in 1.314174838s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1
19702024-07-26T02:38:14.644ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23") failed: Property not found. retry in 516.190027ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23
19712024-07-26T02:38:14.667ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087") failed: Property not found. retry in 55.289096ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087
19722024-07-26T02:38:14.677ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e") failed: Property not found. retry in 47.534541ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e
19732024-07-26T02:38:14.690ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087") failed: Property not found. retry in 54.173804ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087
19742024-07-26T02:38:14.709ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f") failed: Property not found. retry in 911.950162ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f
19752024-07-26T02:38:14.725ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5") failed: Property not found. retry in 1.015841794s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5
19762024-07-26T02:38:14.734ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087") failed: Property not found. retry in 77.135721ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087
19772024-07-26T02:38:14.743ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e") failed: Property not found. retry in 125.308535ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e
19782024-07-26T02:38:14.755ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087") failed: Property not found. retry in 83.188287ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087
19792024-07-26T02:38:14.822ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087") failed: Property not found. retry in 199.491411ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087
19802024-07-26T02:38:14.843ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e") failed: Property not found. retry in 700.662958ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e
19812024-07-26T02:38:14.857ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087") failed: Property not found. retry in 268.100432ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087
19822024-07-26T02:38:14.886ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e") failed: Property not found. retry in 283.666092ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e
19832024-07-26T02:38:14.998ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c") failed: Property not found. retry in 617.715149ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c
19842024-07-26T02:38:15.022ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3") failed: Property not found. retry in 1.333748753s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3
19852024-07-26T02:38:15.034ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087") failed: Property not found. retry in 571.29594ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087
19862024-07-26T02:38:15.062ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c") failed: Property not found. retry in 571.491964ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c
19872024-07-26T02:38:15.139ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087") failed: Property not found. retry in 206.764445ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087
19882024-07-26T02:38:15.162ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f") failed: Property not found. retry in 973.037101ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f
19892024-07-26T02:38:15.173ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23") failed: Property not found. retry in 1.307068184s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23
19902024-07-26T02:38:15.183ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e") failed: Property not found. retry in 599.598136ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e
19912024-07-26T02:38:15.529ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087") failed: Property not found. retry in 441.910557ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087
19922024-07-26T02:38:15.547ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621") failed: Property not found. retry in 1.495780293s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621
19932024-07-26T02:38:15.563ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c") failed: Property not found. retry in 700.10068ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c
19942024-07-26T02:38:15.581ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e") failed: Property not found. retry in 502.666207ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e
19952024-07-26T02:38:15.636ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d") failed: Property not found. retry in 1.089230904s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d
19962024-07-26T02:38:15.654ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f") failed: Property not found. retry in 1.134026384s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f
19972024-07-26T02:38:15.670ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27") failed: Property not found. retry in 1.191226292s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27
19982024-07-26T02:38:15.687ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c") failed: Property not found. retry in 1.092963024s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c
19992024-07-26T02:38:15.701ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087") failed: Property not found. retry in 750.467757ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087
20002024-07-26T02:38:15.714ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c") failed: Property not found. retry in 759.175374ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c
20012024-07-26T02:38:15.755ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5") failed: Property not found. retry in 1.26755065s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5
20022024-07-26T02:38:15.797ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e") failed: Property not found. retry in 1.10489113s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e
20032024-07-26T02:38:15.828ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6") failed: Property not found. retry in 1.22240609s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6
20042024-07-26T02:38:15.938ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1") failed: Property not found. retry in 1.202442041s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1
20052024-07-26T02:38:15.986ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087") failed: Property not found. retry in 980.779676ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087
20062024-07-26T02:38:16.102ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e") failed: Property not found. retry in 723.669448ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e
20072024-07-26T02:38:16.134ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService26'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::8'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte1'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::8]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.5:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
20082024-07-26T02:38:16.134ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425
20092024-07-26T02:38:16.157ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f") failed: Property not found. retry in 858.165153ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f
20102024-07-26T02:38:16.278ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c") failed: Property not found. retry in 1.468255589s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c
20112024-07-26T02:38:16.372ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3") failed: Property not found. retry in 911.433337ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3
20122024-07-26T02:38:16.402ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425") failed: Property not found. retry in 54.974384ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425
20132024-07-26T02:38:16.421ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:61217
20142024-07-26T02:38:16.428ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 6025 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:61217 req_id = 6b11bdfb-2909-42bc-95f1-1658b271dffb response_code = 200 uri = /baseboard
20152024-07-26T02:38:16.465ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087") failed: Property not found. retry in 1.196137983s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087
20162024-07-26T02:38:16.474ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425") failed: Property not found. retry in 110.966544ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425
20172024-07-26T02:38:16.486ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c") failed: Property not found. retry in 1.038957119s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c
20182024-07-26T02:38:16.503ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23") failed: Property not found. retry in 596.922838ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23
20192024-07-26T02:38:16.610ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425") failed: Property not found. retry in 249.449886ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425
20202024-07-26T02:38:16.738ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d") failed: Property not found. retry in 885.682331ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d
20212024-07-26T02:38:16.792ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c") failed: Property not found. retry in 635.811009ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c
20222024-07-26T02:38:16.805ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f") failed: Property not found. retry in 1.316509253s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f
20232024-07-26T02:38:16.847ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e") failed: Property not found. retry in 780.095482ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e
20242024-07-26T02:38:16.900ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425") failed: Property not found. retry in 429.419806ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425
20252024-07-26T02:38:16.922ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27") failed: Property not found. retry in 1.484845151s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27
20262024-07-26T02:38:16.937ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e") failed: Property not found. retry in 1.4148461s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e
20272024-07-26T02:38:16.980ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087") failed: Property not found. retry in 1.164696839s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087
20282024-07-26T02:38:17.030ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f") failed: Property not found. retry in 1.466949806s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f
20292024-07-26T02:38:17.043ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5") failed: Property not found. retry in 948.950213ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5
20302024-07-26T02:38:17.063ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621") failed: Property not found. retry in 1.420350353s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621
20312024-07-26T02:38:17.081ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6") failed: Property not found. retry in 503.780855ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6
20322024-07-26T02:38:17.117ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23") failed: Property not found. retry in 1.038997676s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23
20332024-07-26T02:38:17.155ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1") failed: Property not found. retry in 690.450748ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1
20342024-07-26T02:38:17.294ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3") failed: Property not found. retry in 1.28035267s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3
20352024-07-26T02:38:17.340ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425") failed: Property not found. retry in 984.582926ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425
20362024-07-26T02:38:17.439ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c") failed: Property not found. retry in 1.145139534s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c
20372024-07-26T02:38:17.537ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c") failed: Property not found. retry in 746.331141ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c
20382024-07-26T02:38:17.595ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6") failed: Property not found. retry in 1.359587617s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6
20392024-07-26T02:38:17.634ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d") failed: Property not found. retry in 735.26922ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d
20402024-07-26T02:38:17.643ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e") failed: Property not found. retry in 758.025116ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e
20412024-07-26T02:38:17.671ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087") failed: Property not found. retry in 842.246859ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087
20422024-07-26T02:38:17.756ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c") failed: Property not found. retry in 1.060694638s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c
20432024-07-26T02:38:17.856ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1") failed: Property not found. retry in 709.937301ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1
20442024-07-26T02:38:18.003ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5") failed: Property not found. retry in 859.413568ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5
20452024-07-26T02:38:18.132ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f") failed: Property not found. retry in 535.783487ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f
20462024-07-26T02:38:18.156ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087") failed: Property not found. retry in 1.196678349s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087
20472024-07-26T02:38:18.179ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23") failed: Property not found. retry in 664.495066ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23
20482024-07-26T02:38:18.294ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c") failed: Property not found. retry in 889.919098ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c
20492024-07-26T02:38:18.335ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425") failed: Property not found. retry in 644.780072ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425
20502024-07-26T02:38:18.361ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e") failed: Property not found. retry in 896.317074ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e
20512024-07-26T02:38:18.379ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d") failed: Property not found. retry in 853.835298ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d
20522024-07-26T02:38:18.412ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e") failed: Property not found. retry in 775.515228ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e
20532024-07-26T02:38:18.423ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27") failed: Property not found. retry in 607.460536ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27
20542024-07-26T02:38:18.493ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621") failed: Property not found. retry in 993.421556ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621
20552024-07-26T02:38:18.505ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f") failed: Property not found. retry in 844.824562ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f
20562024-07-26T02:38:18.522ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087") failed: Property not found. retry in 835.775124ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087
20572024-07-26T02:38:18.576ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1") failed: Property not found. retry in 807.663226ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1
20582024-07-26T02:38:18.583ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3") failed: Property not found. retry in 892.108672ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3
20592024-07-26T02:38:18.593ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c") failed: Property not found. retry in 913.859375ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c
20602024-07-26T02:38:18.677ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f") failed: Property not found. retry in 1.376684481s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f
20612024-07-26T02:38:18.831ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c") failed: Property not found. retry in 965.107967ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c
20622024-07-26T02:38:18.858ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23") failed: Property not found. retry in 1.249514832s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23
20632024-07-26T02:38:18.875ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5") failed: Property not found. retry in 725.377256ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5
20642024-07-26T02:38:18.967ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6") failed: Property not found. retry in 1.286571892s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6
20652024-07-26T02:38:18.992ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425") failed: Property not found. retry in 1.422678562s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425
20662024-07-26T02:38:19.043ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27") failed: Property not found. retry in 840.411529ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27
20672024-07-26T02:38:19.199ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c") failed: Property not found. retry in 697.893672ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c
20682024-07-26T02:38:19.211ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e") failed: Property not found. retry in 1.196194513s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e
20692024-07-26T02:38:19.247ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d") failed: Property not found. retry in 1.052716839s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d
20702024-07-26T02:38:19.271ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e") failed: Property not found. retry in 532.236256ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e
20712024-07-26T02:38:19.335ZINFOSledAgent (ServiceManager): Profile for oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService15'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::e'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::e'/> <propval type="astring" name="listen_port" value='8123'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
20722024-07-26T02:38:19.335ZINFOSledAgent (ServiceManager): Booting oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626
20732024-07-26T02:38:19.364ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f") failed: Property not found. retry in 1.459445731s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f
20742024-07-26T02:38:19.377ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087") failed: Property not found. retry in 1.305111251s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087
20752024-07-26T02:38:19.388ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087") failed: Property not found. retry in 719.557575ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087
20762024-07-26T02:38:19.406ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1") failed: Property not found. retry in 1.138770345s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1
20772024-07-26T02:38:19.487ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3") failed: Property not found. retry in 935.804181ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3
20782024-07-26T02:38:19.500ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621") failed: Property not found. retry in 1.39537767s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621
20792024-07-26T02:38:19.524ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c") failed: Property not found. retry in 1.44383283s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c
20802024-07-26T02:38:19.562ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626") failed: Property not found. retry in 37.740104ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626
20812024-07-26T02:38:19.610ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626") failed: Property not found. retry in 111.375532ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626
20822024-07-26T02:38:19.619ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5") failed: Property not found. retry in 1.313422206s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5
20832024-07-26T02:38:19.819ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626") failed: Property not found. retry in 242.576827ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626
20842024-07-26T02:38:19.853ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c") failed: Property not found. retry in 1.428264671s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c
20852024-07-26T02:38:19.867ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e") failed: Property not found. retry in 1.262536331s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e
20862024-07-26T02:38:19.897ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27") failed: Property not found. retry in 646.050198ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27
20872024-07-26T02:38:19.911ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c") failed: Property not found. retry in 1.250749465s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c
20882024-07-26T02:38:20.065ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f") failed: Property not found. retry in 653.58917ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f
20892024-07-26T02:38:20.074ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626") failed: Property not found. retry in 594.263395ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626
20902024-07-26T02:38:20.117ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23") failed: Property not found. retry in 986.617776ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23
20912024-07-26T02:38:20.126ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087") failed: Property not found. retry in 981.426939ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087
20922024-07-26T02:38:20.267ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6") failed: Property not found. retry in 1.393869111s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6
20932024-07-26T02:38:20.309ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d") failed: Property not found. retry in 826.351109ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d
20942024-07-26T02:38:20.417ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e") failed: Property not found. retry in 667.929668ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e
20952024-07-26T02:38:20.426ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425") failed: Property not found. retry in 557.359341ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425
20962024-07-26T02:38:20.436ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3") failed: Property not found. retry in 867.19502ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3
20972024-07-26T02:38:20.554ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27") failed: Property not found. retry in 1.092237915s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27
20982024-07-26T02:38:20.565ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1") failed: Property not found. retry in 1.299264507s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1
20992024-07-26T02:38:20.682ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626") failed: Property not found. retry in 1.083926469s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626
21002024-07-26T02:38:20.691ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087") failed: Property not found. retry in 770.086006ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087
21012024-07-26T02:38:20.732ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f") failed: Property not found. retry in 1.390809443s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f
21022024-07-26T02:38:20.842ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f") failed: Property not found. retry in 696.565888ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f
21032024-07-26T02:38:20.910ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621") failed: Property not found. retry in 1.346052844s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621
21042024-07-26T02:38:20.943ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5") failed: Property not found. retry in 554.072078ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5
21052024-07-26T02:38:20.984ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c") failed: Property not found. retry in 1.06482612s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c
21062024-07-26T02:38:21.001ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425") failed: Property not found. retry in 855.830191ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425
21072024-07-26T02:38:21.099ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e") failed: Property not found. retry in 781.893403ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e
21082024-07-26T02:38:21.118ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23") failed: Property not found. retry in 1.341537553s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23
21092024-07-26T02:38:21.132ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087") failed: Property not found. retry in 750.945395ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087
21102024-07-26T02:38:21.144ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e") failed: Property not found. retry in 950.712549ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e
21112024-07-26T02:38:21.156ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d") failed: Property not found. retry in 1.087172881s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d
21122024-07-26T02:38:21.172ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c") failed: Property not found. retry in 608.581336ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c
21132024-07-26T02:38:21.294ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c") failed: Property not found. retry in 797.983157ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c
21142024-07-26T02:38:21.316ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3") failed: Property not found. retry in 735.889379ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3
21152024-07-26T02:38:21.474ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087") failed: Property not found. retry in 1.074853337s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087
21162024-07-26T02:38:21.510ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5") failed: Property not found. retry in 1.080246636s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5
21172024-07-26T02:38:21.551ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f") failed: Property not found. retry in 1.094737257s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f
21182024-07-26T02:38:21.658ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27") failed: Property not found. retry in 507.402913ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27
21192024-07-26T02:38:21.673ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6") failed: Property not found. retry in 1.131050331s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6
21202024-07-26T02:38:21.778ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626") failed: Property not found. retry in 1.369405875s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626
21212024-07-26T02:38:21.799ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c") failed: Property not found. retry in 873.157157ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c
21222024-07-26T02:38:21.868ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425") failed: Property not found. retry in 612.710816ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425
21232024-07-26T02:38:21.880ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1") failed: Property not found. retry in 1.085287475s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1
21242024-07-26T02:38:21.898ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e") failed: Property not found. retry in 1.131818642s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e
21252024-07-26T02:38:21.908ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087") failed: Property not found. retry in 585.06214ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087
21262024-07-26T02:38:22.073ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c") failed: Property not found. retry in 1.417752912s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c
21272024-07-26T02:38:22.091ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3") failed: Property not found. retry in 571.733707ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3
21282024-07-26T02:38:22.116ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c") failed: Property not found. retry in 1.057008939s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c
21292024-07-26T02:38:22.143ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e") failed: Property not found. retry in 581.239017ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e
21302024-07-26T02:38:22.166ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f") failed: Property not found. retry in 1.287894678s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f
21312024-07-26T02:38:22.186ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27") failed: Property not found. retry in 1.061535996s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27
21322024-07-26T02:38:22.265ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d") failed: Property not found. retry in 860.785529ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d
21332024-07-26T02:38:22.289ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621") failed: Property not found. retry in 1.063532381s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621
21342024-07-26T02:38:22.474ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23") failed: Property not found. retry in 630.91332ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23
21352024-07-26T02:38:22.500ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425") failed: Property not found. retry in 1.24689339s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425
21362024-07-26T02:38:22.516ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087") failed: Property not found. retry in 761.627155ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087
21372024-07-26T02:38:22.566ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087") failed: Property not found. retry in 1.448967794s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087
21382024-07-26T02:38:22.608ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5") failed: Property not found. retry in 928.618566ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5
21392024-07-26T02:38:22.665ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f") failed: Property not found. retry in 1.398402249s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f
21402024-07-26T02:38:22.676ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3") failed: Property not found. retry in 970.858464ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3
21412024-07-26T02:38:22.691ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c") failed: Property not found. retry in 552.72897ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c
21422024-07-26T02:38:22.742ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e") failed: Property not found. retry in 1.246253448s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e
21432024-07-26T02:38:22.820ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6") failed: Property not found. retry in 710.79563ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6
21442024-07-26T02:38:22.980ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1") failed: Property not found. retry in 1.330003901s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1
21452024-07-26T02:38:23.045ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e") failed: Property not found. retry in 776.756091ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e
21462024-07-26T02:38:23.121ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23") failed: Property not found. retry in 1.255162178s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23
21472024-07-26T02:38:23.140ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d") failed: Property not found. retry in 631.217895ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d
21482024-07-26T02:38:23.162ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626") failed: Property not found. retry in 929.111352ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626
21492024-07-26T02:38:23.194ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c") failed: Property not found. retry in 1.051856683s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c
21502024-07-26T02:38:23.259ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c") failed: Property not found. retry in 785.469626ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c
21512024-07-26T02:38:23.273ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27") failed: Property not found. retry in 1.239008194s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27
21522024-07-26T02:38:23.293ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087") failed: Property not found. retry in 540.229297ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087
21532024-07-26T02:38:23.368ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621") failed: Property not found. retry in 1.229397918s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621
21542024-07-26T02:38:23.471ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f") failed: Property not found. retry in 1.125697455s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f
21552024-07-26T02:38:23.505ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c") failed: Property not found. retry in 1.148853328s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c
21562024-07-26T02:38:23.546ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6") failed: Property not found. retry in 1.05879238s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6
21572024-07-26T02:38:23.559ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5") failed: Property not found. retry in 1.448035711s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5
21582024-07-26T02:38:23.663ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3") failed: Property not found. retry in 563.735643ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3
21592024-07-26T02:38:23.766ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425") failed: Property not found. retry in 933.78244ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425
21602024-07-26T02:38:23.787ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d") failed: Property not found. retry in 1.372499075s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d
21612024-07-26T02:38:23.838ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e") failed: Property not found. retry in 1.074200196s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e
21622024-07-26T02:38:23.850ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087") failed: Property not found. retry in 1.30426028s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087
21632024-07-26T02:38:24.004ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e") failed: Property not found. retry in 1.103708218s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e
21642024-07-26T02:38:24.030ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087") failed: Property not found. retry in 1.183857099s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087
21652024-07-26T02:38:24.058ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c") failed: Property not found. retry in 657.82356ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c
21662024-07-26T02:38:24.080ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f") failed: Property not found. retry in 956.872623ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f
21672024-07-26T02:38:24.107ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626") failed: Property not found. retry in 1.363408598s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626
21682024-07-26T02:38:24.244ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3") failed: Property not found. retry in 888.233221ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3
21692024-07-26T02:38:24.263ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c") failed: Property not found. retry in 602.795563ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c
21702024-07-26T02:38:24.321ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1") failed: Property not found. retry in 571.683818ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1
21712024-07-26T02:38:24.390ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23") failed: Property not found. retry in 1.257850719s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23
21722024-07-26T02:38:24.523ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27") failed: Property not found. retry in 623.709386ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27
21732024-07-26T02:38:24.609ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f") failed: Property not found. retry in 1.092516188s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f
21742024-07-26T02:38:24.618ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621") failed: Property not found. retry in 971.714061ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621
21752024-07-26T02:38:24.628ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6") failed: Property not found. retry in 908.962739ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6
21762024-07-26T02:38:24.666ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c") failed: Property not found. retry in 1.232566307s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c
21772024-07-26T02:38:24.712ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425") failed: Property not found. retry in 1.470427323s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425
21782024-07-26T02:38:24.728ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c") failed: Property not found. retry in 654.284243ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c
21792024-07-26T02:38:24.880ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c") failed: Property not found. retry in 947.114385ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c
21802024-07-26T02:38:24.911ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1") failed: Property not found. retry in 1.001180436s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1
21812024-07-26T02:38:24.926ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e") failed: Property not found. retry in 652.197491ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e
21822024-07-26T02:38:25.019ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5") failed: Property not found. retry in 913.500077ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5
21832024-07-26T02:38:25.048ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f") failed: Property not found. retry in 817.553331ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f
21842024-07-26T02:38:25.122ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e") failed: Property not found. retry in 831.412638ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e
21852024-07-26T02:38:25.144ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3") failed: Property not found. retry in 730.890884ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3
21862024-07-26T02:38:25.159ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27") failed: Property not found. retry in 1.095517108s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27
21872024-07-26T02:38:25.168ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087") failed: Property not found. retry in 873.484632ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087
21882024-07-26T02:38:25.180ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d") failed: Property not found. retry in 1.016962993s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d
21892024-07-26T02:38:25.225ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087") failed: Property not found. retry in 1.289672256s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087
21902024-07-26T02:38:25.395ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c") failed: Property not found. retry in 988.581379ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c
21912024-07-26T02:38:25.484ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626") failed: Property not found. retry in 838.881866ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626
21922024-07-26T02:38:25.602ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e") failed: Property not found. retry in 918.820228ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e
21932024-07-26T02:38:25.613ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621") failed: Property not found. retry in 961.825284ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621
21942024-07-26T02:38:25.657ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23") failed: Property not found. retry in 888.882175ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23
21952024-07-26T02:38:25.711ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f") failed: Property not found. retry in 755.076887ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f
21962024-07-26T02:38:25.876ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f") failed: Property not found. retry in 638.341542ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f
21972024-07-26T02:38:25.885ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3") failed: Property not found. retry in 1.377837355s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3
21982024-07-26T02:38:25.912ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c") failed: Property not found. retry in 614.207066ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c
21992024-07-26T02:38:25.922ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1") failed: Property not found. retry in 1.338791258s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1
22002024-07-26T02:38:25.944ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5") failed: Property not found. retry in 954.453482ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5
22012024-07-26T02:38:25.962ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e") failed: Property not found. retry in 635.620711ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e
22022024-07-26T02:38:26.052ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087") failed: Property not found. retry in 1.071010764s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087
22032024-07-26T02:38:26.194ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425") failed: Property not found. retry in 933.621817ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425
22042024-07-26T02:38:26.208ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d") failed: Property not found. retry in 1.249449624s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d
22052024-07-26T02:38:26.265ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27") failed: Property not found. retry in 1.45493989s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27
22062024-07-26T02:38:26.334ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626") failed: Property not found. retry in 680.951464ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626
22072024-07-26T02:38:26.395ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c") failed: Property not found. retry in 1.453915398s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c
22082024-07-26T02:38:26.477ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f") failed: Property not found. retry in 1.180273814s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f
22092024-07-26T02:38:26.526ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f") failed: Property not found. retry in 1.157508516s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f
22102024-07-26T02:38:26.537ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e") failed: Property not found. retry in 866.546369ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e
22112024-07-26T02:38:26.549ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087") failed: Property not found. retry in 1.478837272s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087
22122024-07-26T02:38:26.563ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23") failed: Property not found. retry in 1.199145207s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23
22132024-07-26T02:38:26.575ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c") failed: Property not found. retry in 1.075607011s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c
22142024-07-26T02:38:26.590ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621") failed: Property not found. retry in 1.343455851s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621
22152024-07-26T02:38:26.611ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e") failed: Property not found. retry in 986.387792ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e
22162024-07-26T02:38:27.025ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626") failed: Property not found. retry in 824.604189ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626
22172024-07-26T02:38:27.149ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425") failed: Property not found. retry in 1.134148103s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425
22182024-07-26T02:38:27.895ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626") failed: Property not found. retry in 1.373121892s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626
22192024-07-26T02:38:28.282ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 86.666485233s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error trying to connect: tcp connect error: Connection refused (os error 146)" file = oximeter/producer/src/lib.rs:424 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
22202024-07-26T02:38:29.285ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626") failed: Property not found. retry in 674.464092ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626
22212024-07-26T02:38:29.974ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626") failed: Property not found. retry in 1.392860821s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626
22222024-07-26T02:38:31.378ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626") failed: Property not found. retry in 1.136275398s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626
22232024-07-26T02:38:32.525ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626") failed: Property not found. retry in 1.100014408s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626
22242024-07-26T02:38:33.636ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626") failed: Property not found. retry in 1.127790409s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626
22252024-07-26T02:38:34.776ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626") failed: Property not found. retry in 1.277560012s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626
22262024-07-26T02:38:36.066ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626") failed: Property not found. retry in 1.054270165s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626
22272024-07-26T02:38:37.131ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626") failed: Property not found. retry in 1.096816305s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626
22282024-07-26T02:38:38.239ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626") failed: Property not found. retry in 1.108666427s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626
22292024-07-26T02:38:39.377ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3058 zone = oxz_crucible_07eec7c5-c6b0-4973-a922-2fdc5f972621
22302024-07-26T02:38:39.378ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_cf7587f8-33d7-4e89-bad7-ae8cdd556425
22312024-07-26T02:38:39.378ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_2cc93cb4-8546-4d34-af83-babf2273221c
22322024-07-26T02:38:39.378ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3058 zone = oxz_crucible_7832441b-d9a6-4b73-a71f-8fb8e621ab9f
22332024-07-26T02:38:39.378ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3058 zone = oxz_nexus_56f1d4b9-214d-4104-bc80-a770b9be089e
22342024-07-26T02:38:39.378ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1dcc3b27-f3d4-4d4b-b4c7-1bf76f0e0e23
22352024-07-26T02:38:39.378ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3058 zone = oxz_crucible_2fa8c446-5b0d-456d-a750-28a177c9ab27
22362024-07-26T02:38:39.378ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3058 zone = oxz_clickhouse_d6874d80-d20e-408e-91a2-c5a467770626
22372024-07-26T02:38:39.378ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_290da25c-6635-4875-bb74-2c418e3e33a5
22382024-07-26T02:38:39.378ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3058 zone = oxz_oximeter_8752c2d7-36d8-49ed-860d-ec196a32b72c
22392024-07-26T02:38:39.378ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_53979846-b429-4bae-9968-d96ff860d07e
22402024-07-26T02:38:39.378ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3058 zone = oxz_nexus_9d80bc64-2ec6-4d4e-96d7-5c012d075087
22412024-07-26T02:38:39.378ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3058 zone = oxz_crucible_0a17296b-a160-4ef4-9cb7-01d58322092f
22422024-07-26T02:38:39.378ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3058 zone = oxz_crucible_b56f7823-5e5f-4958-a8ee-f08cb6a361c1
22432024-07-26T02:38:39.378ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3058 zone = oxz_crucible_39533c6a-2c55-4972-8616-1a19033efc8c
22442024-07-26T02:38:39.378ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3058 zone = oxz_crucible_2fc14079-6244-4be0-84a1-1c701868ee2d
22452024-07-26T02:38:39.378ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3058 zone = oxz_crucible_636d501d-962a-441b-8898-3876ad1cb8b3
22462024-07-26T02:38:39.382ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_78641540-4174-48c6-892e-29962a5829c6
22472024-07-26T02:38:39.382ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3058 zone = oxz_nexus_8b4031b8-df71-45f8-b977-b6aa00214087
22482024-07-26T02:38:39.382ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
22492024-07-26T02:38:39.386ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
22502024-07-26T02:38:39.391ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 39724532 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:62928 req_id = 7b78f765-bc5c-4be0-becf-e806b992c041 response_code = 204 uri = /omicron-zones
22512024-07-26T02:38:39.412ZINFOSledAgent (RSS): Finished setting up services file = sled-agent/src/rack_setup/service.rs:1250
22522024-07-26T02:38:39.413ZINFOSledAgent (RSS): Writing ledger to /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
22532024-07-26T02:38:39.413ZINFOSledAgent (RSS): Writing ledger to /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
22542024-07-26T02:38:39.421ZINFOSledAgent (RSS): Handing off control to Nexus file = sled-agent/src/rack_setup/service.rs:687
22552024-07-26T02:38:39.428ZINFOSledAgent (RSS): Nexus address: [fd00:1122:3344:101::b]:12221 file = sled-agent/src/rack_setup/service.rs:700
22562024-07-26T02:38:39.462ZINFOSledAgent (RSS): rack_network_config: RackNetworkConfigV2 { bfd: [], bgp: [], infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV2 { addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.2.174, width: 24, }, ), vlan_id: None, }, ], autoneg: false, bgp_peers: [], port: "qsfp0", routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.2.100, vlan_id: None, }, ], switch: Switch0, uplink_port_fec: None, uplink_port_speed: Speed40G, }, ], rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, } file = sled-agent/src/rack_setup/service.rs:831
22572024-07-26T02:38:39.469ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/6101b963-ee1b-430c-ad60-5a5922d708b7/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
22582024-07-26T02:38:39.573ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/6101b963-ee1b-430c-ad60-5a5922d708b7/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
22592024-07-26T02:38:39.707ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/6101b963-ee1b-430c-ad60-5a5922d708b7/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
22602024-07-26T02:38:39.837ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/6101b963-ee1b-430c-ad60-5a5922d708b7/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
22612024-07-26T02:38:39.939ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/6101b963-ee1b-430c-ad60-5a5922d708b7/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
22622024-07-26T02:38:40.440ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "a378e554-cc84-4c79-8a66-24c0f5848115", "content-length": "173", "date": "Fri, 26 Jul 2024 02:38:40 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: rack with id \"6101b963-ee1b-430c-ad60-5a5922d708b7\"", request_id: "a378e554-cc84-4c79-8a66-24c0f5848115" } file = sled-agent/src/rack_setup/service.rs:893
22632024-07-26T02:38:41.051ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "949a7fad-f12d-4a6a-96b2-992bfa2d2f28", "content-length": "173", "date": "Fri, 26 Jul 2024 02:38:41 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: rack with id \"6101b963-ee1b-430c-ad60-5a5922d708b7\"", request_id: "949a7fad-f12d-4a6a-96b2-992bfa2d2f28" } file = sled-agent/src/rack_setup/service.rs:893
22642024-07-26T02:38:41.276ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "532e50af-9a0d-44d1-a632-4d7b3cfb177b", "content-length": "173", "date": "Fri, 26 Jul 2024 02:38:40 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: rack with id \"6101b963-ee1b-430c-ad60-5a5922d708b7\"", request_id: "532e50af-9a0d-44d1-a632-4d7b3cfb177b" } file = sled-agent/src/rack_setup/service.rs:893
22652024-07-26T02:38:42.651ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "e9b72a64-9bf4-4dd3-8a12-8e0643fff4e1", "content-length": "173", "date": "Fri, 26 Jul 2024 02:38:42 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"ea0e0fc4-fc93-4de6-b687-98567d824e83\"", request_id: "e9b72a64-9bf4-4dd3-8a12-8e0643fff4e1" } file = sled-agent/src/rack_setup/service.rs:893
22662024-07-26T02:38:43.588ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "ea7cb684-b721-41be-91eb-a876df47456b", "content-length": "173", "date": "Fri, 26 Jul 2024 02:38:42 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"ea0e0fc4-fc93-4de6-b687-98567d824e83\"", request_id: "ea7cb684-b721-41be-91eb-a876df47456b" } file = sled-agent/src/rack_setup/service.rs:893
22672024-07-26T02:38:44.303ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "856a373a-8c32-43a0-a562-17c5cbf9bd6d", "content-length": "173", "date": "Fri, 26 Jul 2024 02:38:44 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"ea0e0fc4-fc93-4de6-b687-98567d824e83\"", request_id: "856a373a-8c32-43a0-a562-17c5cbf9bd6d" } file = sled-agent/src/rack_setup/service.rs:893
22682024-07-26T02:38:45.143ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "c1a32466-fcb6-44b3-ad34-6cd5c448ef27", "content-length": "173", "date": "Fri, 26 Jul 2024 02:38:44 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"ea0e0fc4-fc93-4de6-b687-98567d824e83\"", request_id: "c1a32466-fcb6-44b3-ad34-6cd5c448ef27" } file = sled-agent/src/rack_setup/service.rs:893
22692024-07-26T02:38:46.500ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:51809
22702024-07-26T02:38:46.501ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 39 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:51809 req_id = 732670bc-1751-438c-8e2b-91ef39adbf80 response_code = 200 uri = /baseboard
22712024-07-26T02:38:46.521ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "8561b594-00db-4d2e-b8c5-fe3ba07f4584", "content-length": "173", "date": "Fri, 26 Jul 2024 02:38:46 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"ea0e0fc4-fc93-4de6-b687-98567d824e83\"", request_id: "8561b594-00db-4d2e-b8c5-fe3ba07f4584" } file = sled-agent/src/rack_setup/service.rs:893
22722024-07-26T02:38:47.458ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "fe0fc5ad-8be3-4275-b3e6-b9b35bdc21b7", "content-length": "173", "date": "Fri, 26 Jul 2024 02:38:47 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"ea0e0fc4-fc93-4de6-b687-98567d824e83\"", request_id: "fe0fc5ad-8be3-4275-b3e6-b9b35bdc21b7" } file = sled-agent/src/rack_setup/service.rs:893
22732024-07-26T02:38:48.519ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "251cae66-8379-40fe-a0b6-e870e447febc", "content-length": "173", "date": "Fri, 26 Jul 2024 02:38:48 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"ea0e0fc4-fc93-4de6-b687-98567d824e83\"", request_id: "251cae66-8379-40fe-a0b6-e870e447febc" } file = sled-agent/src/rack_setup/service.rs:893
22742024-07-26T02:38:49.774ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "c8c9201c-8a37-4f6b-ba31-e2a970716923", "content-length": "173", "date": "Fri, 26 Jul 2024 02:38:49 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"ea0e0fc4-fc93-4de6-b687-98567d824e83\"", request_id: "c8c9201c-8a37-4f6b-ba31-e2a970716923" } file = sled-agent/src/rack_setup/service.rs:893
22752024-07-26T02:38:52.211ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "40a6c0e3-ae01-48a2-a95a-aabc3fc132ff", "content-length": "173", "date": "Fri, 26 Jul 2024 02:38:52 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"ea0e0fc4-fc93-4de6-b687-98567d824e83\"", request_id: "40a6c0e3-ae01-48a2-a95a-aabc3fc132ff" } file = sled-agent/src/rack_setup/service.rs:893
22762024-07-26T02:38:52.821ZINFOSledAgent (NexusNotifierTask): Successfully put SledAgentInfo to nexus file = sled-agent/src/nexus.rs:573 new_generation = 1 old_generation = 1 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
22772024-07-26T02:38:52.913ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33771
22782024-07-26T02:38:52.915ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
22792024-07-26T02:38:53.009ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 95285 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:33771 req_id = 3dc74787-9f50-40d0-8bf5-6aac9018dec5 response_code = 200 uri = /inventory
22802024-07-26T02:38:53.010ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
22812024-07-26T02:38:53.010ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
22822024-07-26T02:38:53.015ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
22832024-07-26T02:38:53.017ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 7162 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:33771 req_id = d62ba17a-e048-4b10-9133-11c68fb5a0ea response_code = 200 uri = /omicron-zones
22842024-07-26T02:38:55.643ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:63957
22852024-07-26T02:38:55.646ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:673 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 2f46bf86-4fc6-46a8-b573-bf2fb626f25d, kind: Service { id: cf7587f8-33d7-4e89-bad7-ae8cdd556425 }, name: Name("external-dns-cf7587f8-33d7-4e89-bad7-ae8cdd556425"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 168, 111])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 567092fa-fb08-4940-8635-631e5c908557, kind: Service { id: 53979846-b429-4bae-9968-d96ff860d07e }, name: Name("external-dns-53979846-b429-4bae-9968-d96ff860d07e"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 158, 230])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 824ae30e-19ba-423f-923a-3afa2a59b03f, kind: Service { id: 8b4031b8-df71-45f8-b977-b6aa00214087 }, name: Name("nexus-8b4031b8-df71-45f8-b977-b6aa00214087"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 145])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 95c06611-4118-4e93-84ac-1b68f8ff2a6d, kind: Service { id: 9d80bc64-2ec6-4d4e-96d7-5c012d075087 }, name: Name("nexus-9d80bc64-2ec6-4d4e-96d7-5c012d075087"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 148])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e94e36b7-3a4e-4066-8a04-518f25a7ffdb, kind: Service { id: 56f1d4b9-214d-4104-bc80-a770b9be089e }, name: Name("nexus-56f1d4b9-214d-4104-bc80-a770b9be089e"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 180, 23])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
22862024-07-26T02:38:55.647ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
22872024-07-26T02:38:55.647ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
22882024-07-26T02:38:55.647ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
22892024-07-26T02:38:55.647ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
22902024-07-26T02:38:55.647ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte0" rules = []
22912024-07-26T02:38:55.647ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
22922024-07-26T02:38:55.647ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 4293 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:63957 req_id = d831668c-f727-47ab-9649-28141ecabd55 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
22932024-07-26T02:38:55.650ZINFOSledAgent (RSS): Handoff to Nexus is complete file = sled-agent/src/rack_setup/service.rs:903
22942024-07-26T02:38:55.878ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:52248
22952024-07-26T02:38:55.878ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
22962024-07-26T02:38:55.974ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 95953 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:52248 req_id = 4bc8082d-ee93-4d6c-8cfe-df990454b809 response_code = 200 uri = /inventory
22972024-07-26T02:38:55.974ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
22982024-07-26T02:38:55.974ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/config/omicron-zones.json file = common/src/ledger.rs:177
22992024-07-26T02:38:55.976ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/config/omicron-zones.json file = common/src/ledger.rs:177
23002024-07-26T02:38:55.976ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 1759 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:52248 req_id = 6f20aced-4af3-45ed-a808-fbf657e451f1 response_code = 200 uri = /omicron-zones
23012024-07-26T02:38:56.236ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:55604
23022024-07-26T02:38:56.237ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:673 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 2f46bf86-4fc6-46a8-b573-bf2fb626f25d, kind: Service { id: cf7587f8-33d7-4e89-bad7-ae8cdd556425 }, name: Name("external-dns-cf7587f8-33d7-4e89-bad7-ae8cdd556425"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 168, 111])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 567092fa-fb08-4940-8635-631e5c908557, kind: Service { id: 53979846-b429-4bae-9968-d96ff860d07e }, name: Name("external-dns-53979846-b429-4bae-9968-d96ff860d07e"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 158, 230])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 824ae30e-19ba-423f-923a-3afa2a59b03f, kind: Service { id: 8b4031b8-df71-45f8-b977-b6aa00214087 }, name: Name("nexus-8b4031b8-df71-45f8-b977-b6aa00214087"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 145])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 95c06611-4118-4e93-84ac-1b68f8ff2a6d, kind: Service { id: 9d80bc64-2ec6-4d4e-96d7-5c012d075087 }, name: Name("nexus-9d80bc64-2ec6-4d4e-96d7-5c012d075087"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 148])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e94e36b7-3a4e-4066-8a04-518f25a7ffdb, kind: Service { id: 56f1d4b9-214d-4104-bc80-a770b9be089e }, name: Name("nexus-56f1d4b9-214d-4104-bc80-a770b9be089e"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 180, 23])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
23032024-07-26T02:38:56.237ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
23042024-07-26T02:38:56.237ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
23052024-07-26T02:38:56.237ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
23062024-07-26T02:38:56.237ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
23072024-07-26T02:38:56.237ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte0" rules = []
23082024-07-26T02:38:56.237ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
23092024-07-26T02:38:56.237ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 348 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:55604 req_id = 570aafb1-dac0-4d97-9e7a-97e00b5edacb response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23102024-07-26T02:38:56.304ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:46550
23112024-07-26T02:38:56.305ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:673 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 2f46bf86-4fc6-46a8-b573-bf2fb626f25d, kind: Service { id: cf7587f8-33d7-4e89-bad7-ae8cdd556425 }, name: Name("external-dns-cf7587f8-33d7-4e89-bad7-ae8cdd556425"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 168, 111])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 567092fa-fb08-4940-8635-631e5c908557, kind: Service { id: 53979846-b429-4bae-9968-d96ff860d07e }, name: Name("external-dns-53979846-b429-4bae-9968-d96ff860d07e"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 158, 230])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 824ae30e-19ba-423f-923a-3afa2a59b03f, kind: Service { id: 8b4031b8-df71-45f8-b977-b6aa00214087 }, name: Name("nexus-8b4031b8-df71-45f8-b977-b6aa00214087"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 145])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 95c06611-4118-4e93-84ac-1b68f8ff2a6d, kind: Service { id: 9d80bc64-2ec6-4d4e-96d7-5c012d075087 }, name: Name("nexus-9d80bc64-2ec6-4d4e-96d7-5c012d075087"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 148])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e94e36b7-3a4e-4066-8a04-518f25a7ffdb, kind: Service { id: 56f1d4b9-214d-4104-bc80-a770b9be089e }, name: Name("nexus-56f1d4b9-214d-4104-bc80-a770b9be089e"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 180, 23])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
23122024-07-26T02:38:56.305ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
23132024-07-26T02:38:56.305ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
23142024-07-26T02:38:56.305ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
23152024-07-26T02:38:56.305ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
23162024-07-26T02:38:56.305ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte0" rules = []
23172024-07-26T02:38:56.305ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
23182024-07-26T02:38:56.305ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 983 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:46550 req_id = 842237a5-c872-4f06-975e-a93d62a0aae5 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23192024-07-26T02:38:56.323ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:49705
23202024-07-26T02:38:56.324ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:673 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 2f46bf86-4fc6-46a8-b573-bf2fb626f25d, kind: Service { id: cf7587f8-33d7-4e89-bad7-ae8cdd556425 }, name: Name("external-dns-cf7587f8-33d7-4e89-bad7-ae8cdd556425"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 168, 111])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 567092fa-fb08-4940-8635-631e5c908557, kind: Service { id: 53979846-b429-4bae-9968-d96ff860d07e }, name: Name("external-dns-53979846-b429-4bae-9968-d96ff860d07e"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 158, 230])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 824ae30e-19ba-423f-923a-3afa2a59b03f, kind: Service { id: 8b4031b8-df71-45f8-b977-b6aa00214087 }, name: Name("nexus-8b4031b8-df71-45f8-b977-b6aa00214087"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 145])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 95c06611-4118-4e93-84ac-1b68f8ff2a6d, kind: Service { id: 9d80bc64-2ec6-4d4e-96d7-5c012d075087 }, name: Name("nexus-9d80bc64-2ec6-4d4e-96d7-5c012d075087"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 148])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e94e36b7-3a4e-4066-8a04-518f25a7ffdb, kind: Service { id: 56f1d4b9-214d-4104-bc80-a770b9be089e }, name: Name("nexus-56f1d4b9-214d-4104-bc80-a770b9be089e"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 180, 23])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
23212024-07-26T02:38:56.324ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
23222024-07-26T02:38:56.324ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
23232024-07-26T02:38:56.324ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
23242024-07-26T02:38:56.324ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
23252024-07-26T02:38:56.324ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte0" rules = []
23262024-07-26T02:38:56.324ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
23272024-07-26T02:38:56.324ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 879 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:49705 req_id = 1e0d9487-91b9-4538-a67c-bab76ebd8870 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23282024-07-26T02:39:02.553ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:35969
23292024-07-26T02:39:02.554ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 690 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:35969 req_id = 2e5516a2-f204-468c-91a4-10bdaf0643ef response_code = 200 uri = /vpc-routes
23302024-07-26T02:39:02.621ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 1668 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:35969 req_id = d3a00726-7e8b-4421-aa67-a9c5b4ba774b response_code = 204 uri = /vpc-routes
23312024-07-26T02:39:09.765ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
23322024-07-26T02:39:10.914ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:53905
23332024-07-26T02:39:10.914ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 61 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:53905 req_id = 076c3444-381b-4c9f-8e7e-0d7f79709102 response_code = 200 uri = /vpc-routes
23342024-07-26T02:39:10.922ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 88 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:53905 req_id = 84c3c589-cd20-4763-a187-f3e26c2d0bf2 response_code = 204 uri = /vpc-routes
23352024-07-26T02:39:11.046ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:40573
23362024-07-26T02:39:11.046ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 126 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:40573 req_id = 1918c42e-0d2d-437e-b572-6266caaa40e7 response_code = 200 uri = /v2p
23372024-07-26T02:39:11.084ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50293
23382024-07-26T02:39:11.084ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 25 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50293 req_id = 4d5cc864-c371-42b1-9223-721dc065001b response_code = 200 uri = /vpc-routes
23392024-07-26T02:39:11.101ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 93 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:50293 req_id = 41c6436c-3e28-4837-a474-39d28a84c375 response_code = 204 uri = /vpc-routes
23402024-07-26T02:39:11.147ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44853
23412024-07-26T02:39:11.147ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 626 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44853 req_id = 8575098f-1e6b-4509-bf17-b17008ff801a response_code = 200 uri = /v2p
23422024-07-26T02:39:11.264ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:59495
23432024-07-26T02:39:11.265ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
23442024-07-26T02:39:11.329ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
23452024-07-26T02:39:11.353ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 88848 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:59495 req_id = 4e60e1f4-3252-4f9b-9e20-0c99e1566d4d response_code = 204 uri = /switch-ports
23462024-07-26T02:39:11.353ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:52201
23472024-07-26T02:39:11.353ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
23482024-07-26T02:39:11.375ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
23492024-07-26T02:39:11.400ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 46200 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:52201 req_id = f70b908c-f6c1-4003-b616-021a2780dd91 response_code = 204 uri = /switch-ports
23502024-07-26T02:39:11.402ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 2288 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:52201 req_id = 57249559-05d9-4d09-9192-229bc30d963b response_code = 200 uri = /network-bootstore-config
23512024-07-26T02:39:11.402ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 2380 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:59495 req_id = 61ce9c2f-58fe-4ef5-ba3e-83adc904b8a5 response_code = 200 uri = /network-bootstore-config
23522024-07-26T02:39:11.430ZINFOSledAgent (bootstore): Attempting to update network config with generation: 2, current_generation: 1 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-sock-i86pc
23532024-07-26T02:39:11.430ZINFOSledAgent (bootstore): Writing ledger to /pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
23542024-07-26T02:39:11.431ZINFOSledAgent (bootstore): Writing ledger to /pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
23552024-07-26T02:39:11.432ZINFOSledAgent (bootstore): Broadcasting network config with generation 2 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-sock-i86pc
23562024-07-26T02:39:11.432ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 4637 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:52201 req_id = 764314a6-2403-495c-92fb-8263b05c0a7f response_code = 204 uri = /network-bootstore-config
23572024-07-26T02:39:11.832ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:35973
23582024-07-26T02:39:11.832ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 180 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:35973 req_id = d84aea72-dc34-4a03-8f0c-1d29aa010dbe response_code = 200 uri = /v2p
23592024-07-26T02:39:11.900ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:49242
23602024-07-26T02:39:11.900ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 49 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:49242 req_id = 7d756b68-87cd-4e8e-a660-1f76cc3e39f8 response_code = 200 uri = /vpc-routes
23612024-07-26T02:39:11.915ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 559 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:49242 req_id = b1185115-0ad7-48e1-8462-b7641e9bd20b response_code = 204 uri = /vpc-routes
23622024-07-26T02:39:12.008ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:40088
23632024-07-26T02:39:12.009ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
23642024-07-26T02:39:12.033ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
23652024-07-26T02:39:12.056ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 46747 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:40088 req_id = a92a06f3-edc7-4ecd-9494-2fb6efd5c659 response_code = 204 uri = /switch-ports
23662024-07-26T02:39:12.058ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 108 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:40088 req_id = d744678e-b346-4032-b32e-d9df8258ab33 response_code = 200 uri = /network-bootstore-config
23672024-07-26T02:39:16.534ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:52498
23682024-07-26T02:39:16.534ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 58 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:52498 req_id = 0572b3e0-e0ff-47dc-9e38-ef2008728561 response_code = 200 uri = /baseboard
23692024-07-26T02:39:40.968ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:61861
23702024-07-26T02:39:40.968ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 61 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:61861 req_id = 6fd0790e-caaa-4633-83f7-751a36c5dcff response_code = 200 uri = /vpc-routes
23712024-07-26T02:39:40.982ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37412
23722024-07-26T02:39:40.983ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 812 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:37412 req_id = 9725d9cc-ce61-43f0-a4d2-2975c2120d01 response_code = 200 uri = /vpc-routes
23732024-07-26T02:39:40.992ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 92 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:37412 req_id = ef6039f5-ac7f-4ecd-a0c0-ddd93b64e6e3 response_code = 204 uri = /vpc-routes
23742024-07-26T02:39:41.035ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:53269
23752024-07-26T02:39:41.035ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 176 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:53269 req_id = 6d9e9e60-83a3-44e8-936b-46ec4e47104d response_code = 200 uri = /v2p
23762024-07-26T02:39:41.069ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:46792
23772024-07-26T02:39:41.069ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 702 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:46792 req_id = 5c2c2539-679e-40b1-8498-2adb54c54ff7 response_code = 200 uri = /v2p
23782024-07-26T02:39:41.090ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 84 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:61861 req_id = f93144d1-55ba-4172-bc83-f0afc56c2bd2 response_code = 204 uri = /vpc-routes
23792024-07-26T02:39:41.286ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:65403
23802024-07-26T02:39:41.286ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
23812024-07-26T02:39:41.313ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
23822024-07-26T02:39:41.335ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 49003 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:65403 req_id = 58656701-abe5-44d3-93fd-758d2a0dc04c response_code = 204 uri = /switch-ports
23832024-07-26T02:39:41.335ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 110 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:65403 req_id = 172a9490-e5b3-479f-bb94-e626d3f4b7bc response_code = 200 uri = /network-bootstore-config
23842024-07-26T02:39:41.399ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:42577
23852024-07-26T02:39:41.399ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
23862024-07-26T02:39:41.421ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
23872024-07-26T02:39:41.441ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 42099 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:42577 req_id = a232fb9d-b5ab-44d0-992e-e654049022cd response_code = 204 uri = /switch-ports
23882024-07-26T02:39:41.442ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 128 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:42577 req_id = 28c176eb-1773-4259-bbbc-5a0ec4836efb response_code = 200 uri = /network-bootstore-config
23892024-07-26T02:39:41.995ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62389
23902024-07-26T02:39:41.995ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 78 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:62389 req_id = 2c3e47c4-13e6-4ee4-8068-5ed8c8b198de response_code = 200 uri = /vpc-routes
23912024-07-26T02:39:42.004ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 1137 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:62389 req_id = 4ace6092-dd10-44b0-bed8-0efb3df3d534 response_code = 204 uri = /vpc-routes
23922024-07-26T02:39:42.197ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54734
23932024-07-26T02:39:42.197ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 172 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54734 req_id = dd8b5765-c98d-43de-b7d2-22a69bce9afc response_code = 200 uri = /v2p
23942024-07-26T02:39:42.421ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:49160
23952024-07-26T02:39:42.421ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
23962024-07-26T02:39:42.447ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
23972024-07-26T02:39:42.474ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 52400 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:49160 req_id = 1cb0d33a-3384-42af-8422-1d28190c6029 response_code = 204 uri = /switch-ports
23982024-07-26T02:39:42.474ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:49160 req_id = 9ec93343-2b7a-4909-a07a-452d12a0a6a8 response_code = 200 uri = /network-bootstore-config
23992024-07-26T02:39:46.565ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:46047
24002024-07-26T02:39:46.566ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 61 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:46047 req_id = f779f680-8387-4213-99e6-737225c7241b response_code = 200 uri = /baseboard
24012024-07-26T02:40:10.871ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:54801
24022024-07-26T02:40:10.871ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 109 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:54801 req_id = 302f36f1-099e-41f2-8157-20b7fc1131f1 response_code = 200 uri = /vpc-routes
24032024-07-26T02:40:10.879ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 130 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:54801 req_id = 5e6a7cd4-1a2d-4773-8dc8-94840319c8c5 response_code = 204 uri = /vpc-routes
24042024-07-26T02:40:10.882ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:48181
24052024-07-26T02:40:10.882ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 26 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:48181 req_id = 0818e466-c3f1-4cd3-b205-55799a856d90 response_code = 200 uri = /vpc-routes
24062024-07-26T02:40:10.891ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 810 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:48181 req_id = d833306b-e85b-4c98-8d5b-29a312381a72 response_code = 204 uri = /vpc-routes
24072024-07-26T02:40:10.937ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:38244
24082024-07-26T02:40:10.937ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 107 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:38244 req_id = d299bf68-c77f-411e-888a-5e9f76b1a822 response_code = 200 uri = /v2p
24092024-07-26T02:40:11.039ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:58483
24102024-07-26T02:40:11.039ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 110 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:58483 req_id = 2cef27c0-a4c4-4086-972a-9844cb3b22ae response_code = 200 uri = /v2p
24112024-07-26T02:40:11.089ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:63857
24122024-07-26T02:40:11.089ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
24132024-07-26T02:40:11.116ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
24142024-07-26T02:40:11.138ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 49085 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:63857 req_id = da5699cc-8fc1-4828-b9c5-3ee4fee29c34 response_code = 204 uri = /switch-ports
24152024-07-26T02:40:11.139ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:45610
24162024-07-26T02:40:11.139ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
24172024-07-26T02:40:11.142ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 3175 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:63857 req_id = df612bd1-191d-4587-a2ed-a2bfddd7a223 response_code = 200 uri = /network-bootstore-config
24182024-07-26T02:40:11.161ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
24192024-07-26T02:40:11.181ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 42218 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:45610 req_id = 670a61a9-413d-4f1b-98b6-88132edb0ec3 response_code = 204 uri = /switch-ports
24202024-07-26T02:40:11.181ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 141 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:45610 req_id = 7e971345-1807-476d-90a1-f0a2a9819c88 response_code = 200 uri = /network-bootstore-config
24212024-07-26T02:40:11.880ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:53990
24222024-07-26T02:40:11.881ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 904 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:53990 req_id = 8f2fc300-f3ab-47b9-ba24-3f4f2467ad65 response_code = 200 uri = /v2p
24232024-07-26T02:40:11.916ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:43610
24242024-07-26T02:40:11.916ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 72 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:43610 req_id = 32582941-70ce-4a23-a550-6f670109045f response_code = 200 uri = /vpc-routes
24252024-07-26T02:40:11.924ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:43610 req_id = f0bd3539-5649-400f-a049-d67bfcd9a71c response_code = 204 uri = /vpc-routes
24262024-07-26T02:40:12.066ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:44162
24272024-07-26T02:40:12.066ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
24282024-07-26T02:40:12.088ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
24292024-07-26T02:40:12.111ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 45104 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:44162 req_id = 0b679cc1-ef03-4ce3-838b-8b92376d99ee response_code = 204 uri = /switch-ports
24302024-07-26T02:40:12.112ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 105 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:44162 req_id = 81b50823-3407-49e9-80d9-a6d09ff3127b response_code = 200 uri = /network-bootstore-config
24312024-07-26T02:40:16.599ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:55383
24322024-07-26T02:40:16.599ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 69 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:55383 req_id = f3e9290c-c92c-406c-83fd-0ab0b5809b47 response_code = 200 uri = /baseboard
24332024-07-26T02:40:25.066ZINFOSledAgent (dropshot): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:33355 remote_addr = [fd00:1122:3344:101::d]:44106 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83
24342024-07-26T02:40:25.079ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 12095 local_addr = [fd00:1122:3344:101::1]:33355 method = GET remote_addr = [fd00:1122:3344:101::d]:44106 req_id = 6279636e-e282-4da3-a9af-0bfbdfafb846 response_code = 200 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83 uri = /ea0e0fc4-fc93-4de6-b687-98567d824e83
24352024-07-26T02:40:40.974ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:46049
24362024-07-26T02:40:40.974ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 47 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:46049 req_id = 916f0460-03a9-4168-b26e-775e9e85bb86 response_code = 200 uri = /vpc-routes
24372024-07-26T02:40:40.979ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:43310
24382024-07-26T02:40:40.979ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 18 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:43310 req_id = 096b758c-530e-47df-8c65-c6fa60c108fd response_code = 200 uri = /vpc-routes
24392024-07-26T02:40:40.982ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 82 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:46049 req_id = deec261d-436a-4cbb-abdc-dfb31276df94 response_code = 204 uri = /vpc-routes
24402024-07-26T02:40:40.987ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 41 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:43310 req_id = f31f397b-091f-413b-b0dc-d7570dc5b723 response_code = 204 uri = /vpc-routes
24412024-07-26T02:40:41.130ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:49273
24422024-07-26T02:40:41.130ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 116 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:49273 req_id = d4d683a1-c9dd-4cba-8771-f0febe592b61 response_code = 200 uri = /v2p
24432024-07-26T02:40:41.231ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44485
24442024-07-26T02:40:41.231ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 86 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44485 req_id = 29269c25-77e3-4d7c-acfa-df8b870a8b17 response_code = 200 uri = /v2p
24452024-07-26T02:40:41.316ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:54851
24462024-07-26T02:40:41.317ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
24472024-07-26T02:40:41.344ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
24482024-07-26T02:40:41.368ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 51960 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:54851 req_id = d2d16d2e-1aad-4b10-bd19-cd904c95a655 response_code = 204 uri = /switch-ports
24492024-07-26T02:40:41.368ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:50181
24502024-07-26T02:40:41.368ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
24512024-07-26T02:40:41.392ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
24522024-07-26T02:40:41.414ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 45818 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:50181 req_id = 488dc697-610f-4689-b873-85178bf24005 response_code = 204 uri = /switch-ports
24532024-07-26T02:40:41.414ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 95 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:54851 req_id = dd403d86-7fa4-4d89-a0d9-16714a9f4d42 response_code = 200 uri = /network-bootstore-config
24542024-07-26T02:40:41.415ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 843 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:50181 req_id = 74bdc8fe-8001-405d-b3d7-53119264b84f response_code = 200 uri = /network-bootstore-config
24552024-07-26T02:40:41.874ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:65171
24562024-07-26T02:40:41.875ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 66 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:65171 req_id = ecf13e89-f4df-4e49-8a71-98771a3c699b response_code = 200 uri = /vpc-routes
24572024-07-26T02:40:41.882ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 127 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:65171 req_id = 384877ec-0091-416c-abaa-793f58314c38 response_code = 204 uri = /vpc-routes
24582024-07-26T02:40:41.908ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:55102
24592024-07-26T02:40:41.908ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 101 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:55102 req_id = 7c37fbd2-0ab1-44dd-85f7-d3b29458c657 response_code = 200 uri = /v2p
24602024-07-26T02:40:42.351ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:35063
24612024-07-26T02:40:42.351ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
24622024-07-26T02:40:42.376ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
24632024-07-26T02:40:42.401ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 49927 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:35063 req_id = 43de9385-d0ca-45a3-ba28-534e9a2b9801 response_code = 204 uri = /switch-ports
24642024-07-26T02:40:42.401ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 108 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:35063 req_id = b4ee0ce6-6d30-43d9-841d-ddb8422a674b response_code = 200 uri = /network-bootstore-config
24652024-07-26T02:40:46.633ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:52915
24662024-07-26T02:40:46.633ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 78 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:52915 req_id = 1b416c65-b104-4ff8-aed0-45a0c8c6a0d3 response_code = 200 uri = /baseboard
24672024-07-26T02:40:55.066ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 597 local_addr = [fd00:1122:3344:101::1]:33355 method = GET remote_addr = [fd00:1122:3344:101::d]:44106 req_id = dd2c762e-d39f-48d9-98ba-41e2e6fecdc4 response_code = 200 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83 uri = /ea0e0fc4-fc93-4de6-b687-98567d824e83
24682024-07-26T02:41:10.893ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:50758
24692024-07-26T02:41:10.893ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 98 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:50758 req_id = 23934f21-7583-461e-b65c-255e162016cf response_code = 200 uri = /vpc-routes
24702024-07-26T02:41:10.900ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 115 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:50758 req_id = fddd23a5-ac02-4554-b431-4d13ecec363d response_code = 204 uri = /vpc-routes
24712024-07-26T02:41:10.913ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:55264
24722024-07-26T02:41:10.913ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 125 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:55264 req_id = 9744a675-412e-4e1f-8675-9e9569a3c3c7 response_code = 200 uri = /v2p
24732024-07-26T02:41:10.934ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:57929
24742024-07-26T02:41:10.935ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 908 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:57929 req_id = 6c19d354-7c31-4f3d-97ff-a1d7b11973ab response_code = 200 uri = /v2p
24752024-07-26T02:41:10.976ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:49217
24762024-07-26T02:41:10.976ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 80 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:49217 req_id = b57b6ba8-42fa-4ea0-8743-17580a4f70c9 response_code = 200 uri = /vpc-routes
24772024-07-26T02:41:10.984ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 110 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:49217 req_id = 4c2d101d-c7f0-4cf1-b8ba-8a0d544b326f response_code = 204 uri = /vpc-routes
24782024-07-26T02:41:11.045ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:56173
24792024-07-26T02:41:11.045ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
24802024-07-26T02:41:11.071ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
24812024-07-26T02:41:11.093ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 48662 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:56173 req_id = e699c91f-af57-41e5-ad79-dbf172645162 response_code = 204 uri = /switch-ports
24822024-07-26T02:41:11.094ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:44544
24832024-07-26T02:41:11.094ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
24842024-07-26T02:41:11.119ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
24852024-07-26T02:41:11.143ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 48600 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:44544 req_id = 441c3881-f54b-449c-a9c9-dfdb83aa51bf response_code = 204 uri = /switch-ports
24862024-07-26T02:41:11.143ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 135 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:56173 req_id = f091545e-480e-47a6-bf14-86c9f262cc9e response_code = 200 uri = /network-bootstore-config
24872024-07-26T02:41:11.144ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 667 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:44544 req_id = e807d624-4903-48c5-8f57-db7cea2bf48f response_code = 200 uri = /network-bootstore-config
24882024-07-26T02:41:11.848ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34281
24892024-07-26T02:41:11.848ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 157 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:34281 req_id = dde749c4-2ead-46f8-a2af-0a193bba0947 response_code = 200 uri = /v2p
24902024-07-26T02:41:11.901ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62101
24912024-07-26T02:41:11.901ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 66 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:62101 req_id = bc4e8211-fa0d-487d-a527-bfc1278d8f78 response_code = 200 uri = /vpc-routes
24922024-07-26T02:41:11.909ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 115 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:62101 req_id = f9d5cd20-4cb4-44d2-80f1-80f878ae170d response_code = 204 uri = /vpc-routes
24932024-07-26T02:41:12.062ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:37459
24942024-07-26T02:41:12.062ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
24952024-07-26T02:41:12.085ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
24962024-07-26T02:41:12.109ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 46825 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:37459 req_id = aefcdfa0-c271-4e30-bb1b-87f928a94e02 response_code = 204 uri = /switch-ports
24972024-07-26T02:41:12.109ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 123 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:37459 req_id = 4c15f30f-ce38-46f9-9a32-a2831a35e5a9 response_code = 200 uri = /network-bootstore-config
24982024-07-26T02:41:16.667ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:45078
24992024-07-26T02:41:16.667ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 76 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:45078 req_id = 29a9bad0-0088-4322-8a3e-89887c21fdb8 response_code = 200 uri = /baseboard
25002024-07-26T02:41:25.065ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 528 local_addr = [fd00:1122:3344:101::1]:33355 method = GET remote_addr = [fd00:1122:3344:101::d]:44106 req_id = 6448806a-0a2f-4d62-826d-7642eb73bdeb response_code = 200 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83 uri = /ea0e0fc4-fc93-4de6-b687-98567d824e83
25012024-07-26T02:41:40.961ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:64212
25022024-07-26T02:41:40.962ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 958 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:64212 req_id = 96f74ebb-0824-46dc-af18-9f071fcfe82d response_code = 200 uri = /vpc-routes
25032024-07-26T02:41:40.970ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 113 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:64212 req_id = 1d1692c9-55e1-434a-be4a-6d5d03098ee1 response_code = 204 uri = /vpc-routes
25042024-07-26T02:41:41.109ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:48324
25052024-07-26T02:41:41.109ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 124 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:48324 req_id = 306c0500-49b8-46b7-bdd2-cc87518ecf9e response_code = 200 uri = /v2p
25062024-07-26T02:41:41.127ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:52202
25072024-07-26T02:41:41.127ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 94 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:52202 req_id = daa96d31-9ea0-430e-8940-615e97d8ac44 response_code = 200 uri = /v2p
25082024-07-26T02:41:41.143ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61367
25092024-07-26T02:41:41.143ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 25 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61367 req_id = 2c89e017-8e89-4f0c-b014-ae9e55d1ffb7 response_code = 200 uri = /vpc-routes
25102024-07-26T02:41:41.151ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 79 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:61367 req_id = 7ae047d7-83dd-4d22-8ff6-d84ba71e271f response_code = 204 uri = /vpc-routes
25112024-07-26T02:41:41.294ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:60129
25122024-07-26T02:41:41.294ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
25132024-07-26T02:41:41.316ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
25142024-07-26T02:41:41.339ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 44727 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:60129 req_id = 1d7437fa-3eab-4800-8e55-168f54a2ba23 response_code = 204 uri = /switch-ports
25152024-07-26T02:41:41.339ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37847
25162024-07-26T02:41:41.339ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
25172024-07-26T02:41:41.342ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 3234 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:60129 req_id = 5225e2d5-c2a9-466f-ac9a-d6009fcd810b response_code = 200 uri = /network-bootstore-config
25182024-07-26T02:41:41.364ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
25192024-07-26T02:41:41.386ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 46830 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:37847 req_id = a345ae03-4f4d-4d53-a6f5-0968d93ba831 response_code = 204 uri = /switch-ports
25202024-07-26T02:41:41.387ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 736 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:37847 req_id = 95a53a6e-cf25-4b0f-b790-ebbaa5ee9f57 response_code = 200 uri = /network-bootstore-config
25212024-07-26T02:41:41.857ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:58260
25222024-07-26T02:41:41.857ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 107 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:58260 req_id = 419f7801-f579-43ec-b2a4-3d2c588b0d4c response_code = 200 uri = /vpc-routes
25232024-07-26T02:41:41.865ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 142 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:58260 req_id = 47ddf004-a3a3-49d7-ba87-05e4d440206e response_code = 204 uri = /vpc-routes
25242024-07-26T02:41:41.978ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:53158
25252024-07-26T02:41:41.978ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 825 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:53158 req_id = ee05eddb-e362-4f8e-a37c-648ba69262f1 response_code = 200 uri = /v2p
25262024-07-26T02:41:42.142ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:60069
25272024-07-26T02:41:42.143ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
25282024-07-26T02:41:42.165ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
25292024-07-26T02:41:42.185ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 42930 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:60069 req_id = 32fe988e-8d93-4ec5-8768-8f65822f74f4 response_code = 204 uri = /switch-ports
25302024-07-26T02:41:42.185ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:60069 req_id = 0c77ab8c-4327-4d3a-8413-401b0dfd1752 response_code = 200 uri = /network-bootstore-config
25312024-07-26T02:41:46.701ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:46018
25322024-07-26T02:41:46.701ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 96 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:46018 req_id = 90ec957d-96bd-4e71-b583-46f1db03dc47 response_code = 200 uri = /baseboard
25332024-07-26T02:41:55.065ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 617 local_addr = [fd00:1122:3344:101::1]:33355 method = GET remote_addr = [fd00:1122:3344:101::d]:44106 req_id = 3a95b168-a2f0-45be-9558-4087885ab5c4 response_code = 200 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83 uri = /ea0e0fc4-fc93-4de6-b687-98567d824e83
25342024-07-26T02:42:10.913ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:43399
25352024-07-26T02:42:10.914ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 85 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:43399 req_id = 76e9e2ed-56e8-4b59-b3b8-2cc66f91c9ff response_code = 200 uri = /vpc-routes
25362024-07-26T02:42:10.921ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 110 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:43399 req_id = 135aa72d-157f-4248-92a1-176d82c09ecd response_code = 204 uri = /vpc-routes
25372024-07-26T02:42:10.923ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50633
25382024-07-26T02:42:10.923ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 94 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50633 req_id = f6896eb9-5039-49fe-a39f-9642d0776c14 response_code = 200 uri = /v2p
25392024-07-26T02:42:10.931ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:64173
25402024-07-26T02:42:10.932ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 33 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:64173 req_id = ec59ef0e-42e2-43a0-9541-152053d95489 response_code = 200 uri = /vpc-routes
25412024-07-26T02:42:10.939ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 64 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:64173 req_id = 2dbc4125-84b7-4a25-97e2-12539e4be2c7 response_code = 204 uri = /vpc-routes
25422024-07-26T02:42:11.018ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:33476
25432024-07-26T02:42:11.018ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 88 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:33476 req_id = 2727f9f5-2b53-4686-8e98-ab7a836aad41 response_code = 200 uri = /v2p
25442024-07-26T02:42:11.102ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:39085
25452024-07-26T02:42:11.103ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
25462024-07-26T02:42:11.125ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
25472024-07-26T02:42:11.144ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 41257 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:39085 req_id = 795408f8-4318-4744-92d9-07023b7f2d3f response_code = 204 uri = /switch-ports
25482024-07-26T02:42:11.144ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:39085 req_id = 42a1e23d-6a08-482e-b3ef-5ee71603d5e1 response_code = 200 uri = /network-bootstore-config
25492024-07-26T02:42:11.219ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:53988
25502024-07-26T02:42:11.219ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
25512024-07-26T02:42:11.244ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
25522024-07-26T02:42:11.265ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 45628 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:53988 req_id = a9ba5d2d-f481-4b06-b826-0f34da73316a response_code = 204 uri = /switch-ports
25532024-07-26T02:42:11.265ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 84 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:53988 req_id = 57ce4c1a-1354-414d-9d3a-39d1cd29bc89 response_code = 200 uri = /network-bootstore-config
25542024-07-26T02:42:11.765ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:58722
25552024-07-26T02:42:11.765ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:58722 req_id = 3b6ad828-d0d1-40ac-bcd4-7810b1e90b82 response_code = 200 uri = /vpc-routes
25562024-07-26T02:42:11.773ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 126 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:58722 req_id = 03c82061-d404-483e-bc5e-cb6988337054 response_code = 204 uri = /vpc-routes
25572024-07-26T02:42:11.791ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54815
25582024-07-26T02:42:11.791ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 115 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54815 req_id = 7dca0f7e-0177-4347-a6f2-cd3cea63038d response_code = 200 uri = /v2p
25592024-07-26T02:42:11.895ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:50487
25602024-07-26T02:42:11.896ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
25612024-07-26T02:42:11.920ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
25622024-07-26T02:42:11.941ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 45444 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:50487 req_id = 98a2776e-7f20-4fcd-a819-2362a114196d response_code = 204 uri = /switch-ports
25632024-07-26T02:42:11.941ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 94 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:50487 req_id = b1d2a232-0975-4db6-8b02-dd65b18cbf03 response_code = 200 uri = /network-bootstore-config
25642024-07-26T02:42:16.746ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:42620
25652024-07-26T02:42:16.746ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 56 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:42620 req_id = 469f7fc5-4280-42a2-9a81-c59cea3e65b4 response_code = 200 uri = /baseboard
25662024-07-26T02:42:25.066ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 565 local_addr = [fd00:1122:3344:101::1]:33355 method = GET remote_addr = [fd00:1122:3344:101::d]:44106 req_id = 030f0bf6-fda3-46ad-8d6c-2273e51cfa40 response_code = 200 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83 uri = /ea0e0fc4-fc93-4de6-b687-98567d824e83
25672024-07-26T02:42:40.981ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:48421
25682024-07-26T02:42:40.981ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 54 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:48421 req_id = a79f423e-a9fb-45a6-83a4-385717970c27 response_code = 200 uri = /vpc-routes
25692024-07-26T02:42:40.988ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 94 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:48421 req_id = 657dc0a8-9677-41a3-a37b-ffc152a13484 response_code = 204 uri = /vpc-routes
25702024-07-26T02:42:41.128ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:62987
25712024-07-26T02:42:41.129ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 1362 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:62987 req_id = 6479262e-ad14-408c-bf45-456134022c8e response_code = 200 uri = /v2p
25722024-07-26T02:42:41.142ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:40050
25732024-07-26T02:42:41.142ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 28 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:40050 req_id = 8d525d2e-e0b4-47df-8276-6bfe8acae3e1 response_code = 200 uri = /vpc-routes
25742024-07-26T02:42:41.150ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 84 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:40050 req_id = 8da3ae22-d783-4ddf-957f-05d258c097b3 response_code = 204 uri = /vpc-routes
25752024-07-26T02:42:41.193ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:47977
25762024-07-26T02:42:41.194ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 88 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:47977 req_id = d76851cf-53b6-44c7-a8f7-adfd1e474afb response_code = 200 uri = /v2p
25772024-07-26T02:42:41.296ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:47121
25782024-07-26T02:42:41.296ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
25792024-07-26T02:42:41.321ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
25802024-07-26T02:42:41.344ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 47598 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:47121 req_id = 8f2734d4-c93d-4a6a-90a0-89b95f6cf772 response_code = 204 uri = /switch-ports
25812024-07-26T02:42:41.345ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 739 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:47121 req_id = a6c26da6-e2d5-48f7-8d9e-9ebe15aad411 response_code = 200 uri = /network-bootstore-config
25822024-07-26T02:42:41.382ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50434
25832024-07-26T02:42:41.383ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
25842024-07-26T02:42:41.405ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
25852024-07-26T02:42:41.426ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 43795 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:50434 req_id = 93348065-bc49-4d9b-9c75-bd210a9aa67b response_code = 204 uri = /switch-ports
25862024-07-26T02:42:41.427ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 94 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50434 req_id = d974a21d-84f8-4ce4-a8a1-ca869094a297 response_code = 200 uri = /network-bootstore-config
25872024-07-26T02:42:41.873ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:37520
25882024-07-26T02:42:41.874ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:37520 req_id = 5007455c-4c78-4ca7-8656-970abcfbbe65 response_code = 200 uri = /vpc-routes
25892024-07-26T02:42:41.880ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 137 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:37520 req_id = ea08c726-0830-48ba-93ed-9c53a3a7f126 response_code = 204 uri = /vpc-routes
25902024-07-26T02:42:41.921ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62579
25912024-07-26T02:42:41.921ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 124 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:62579 req_id = 0621b4a7-98a0-47d6-a199-e7cc4202b5f1 response_code = 200 uri = /v2p
25922024-07-26T02:42:42.203ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:48642
25932024-07-26T02:42:42.203ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
25942024-07-26T02:42:42.229ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
25952024-07-26T02:42:42.253ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 49461 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:48642 req_id = b263be18-954c-4fd6-aede-d8952e56d22c response_code = 204 uri = /switch-ports
25962024-07-26T02:42:42.253ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:48642 req_id = 779f0f7b-1953-4dc0-af01-707b574edf1e response_code = 200 uri = /network-bootstore-config
25972024-07-26T02:42:46.781ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:46229
25982024-07-26T02:42:46.781ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 550 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:46229 req_id = 1fde6376-ece6-4702-8801-bfe788631212 response_code = 200 uri = /baseboard
25992024-07-26T02:42:55.121ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 593 local_addr = [fd00:1122:3344:101::1]:33355 method = GET remote_addr = [fd00:1122:3344:101::d]:44106 req_id = 03541c1d-7f11-46f7-84e4-b9132d363aee response_code = 200 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83 uri = /ea0e0fc4-fc93-4de6-b687-98567d824e83
26002024-07-26T02:43:10.913ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:65025
26012024-07-26T02:43:10.914ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 213 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:65025 req_id = f59ccf9e-b9e5-45c6-9f93-583e5cd5d516 response_code = 200 uri = /v2p
26022024-07-26T02:43:10.921ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:47355
26032024-07-26T02:43:10.921ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 67 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:47355 req_id = 98ddd58d-46a5-40e1-91c3-87b4575e7a43 response_code = 200 uri = /vpc-routes
26042024-07-26T02:43:10.928ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 137 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:47355 req_id = 35c79077-9874-4489-95ce-52500ed88a49 response_code = 204 uri = /vpc-routes
26052024-07-26T02:43:10.950ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:49072
26062024-07-26T02:43:10.951ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 111 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:49072 req_id = c905438a-7acd-4c24-9c23-d8dddb3b22d3 response_code = 200 uri = /v2p
26072024-07-26T02:43:11.008ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:47770
26082024-07-26T02:43:11.008ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 27 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:47770 req_id = cb3d502c-10b7-4f38-bf0e-eab87e04da56 response_code = 200 uri = /vpc-routes
26092024-07-26T02:43:11.017ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 74 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:47770 req_id = 70932880-d38b-4f1a-8bcd-aba3ce97e1d0 response_code = 204 uri = /vpc-routes
26102024-07-26T02:43:11.119ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:39650
26112024-07-26T02:43:11.119ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
26122024-07-26T02:43:11.143ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
26132024-07-26T02:43:11.166ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 47509 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:39650 req_id = fedd0c2c-0293-495d-aadc-9da869b57016 response_code = 204 uri = /switch-ports
26142024-07-26T02:43:11.166ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:43407
26152024-07-26T02:43:11.166ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
26162024-07-26T02:43:11.189ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
26172024-07-26T02:43:11.211ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 45048 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:43407 req_id = 13674255-eed5-4146-b40c-364346732ef6 response_code = 204 uri = /switch-ports
26182024-07-26T02:43:11.212ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 117 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:39650 req_id = 8d7b9b4a-d276-4f2c-bfaa-7e81e9bafa3e response_code = 200 uri = /network-bootstore-config
26192024-07-26T02:43:11.212ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 135 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:43407 req_id = 82c07c30-c5d8-46e7-b578-707fb11f7018 response_code = 200 uri = /network-bootstore-config
26202024-07-26T02:43:11.793ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:39477
26212024-07-26T02:43:11.793ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 120 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:39477 req_id = dc572608-d2d7-4a3b-993f-ab8c750e13ca response_code = 200 uri = /vpc-routes
26222024-07-26T02:43:11.800ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 120 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:39477 req_id = 608b930b-508f-48bb-80e5-183edda7ff1c response_code = 204 uri = /vpc-routes
26232024-07-26T02:43:11.837ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57976
26242024-07-26T02:43:11.837ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 145 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:57976 req_id = 16482ab9-a719-42dc-9ae5-cfbac45c5ceb response_code = 200 uri = /v2p
26252024-07-26T02:43:12.002ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:49998
26262024-07-26T02:43:12.003ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
26272024-07-26T02:43:12.030ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
26282024-07-26T02:43:12.053ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 50368 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:49998 req_id = db30d9c9-55ea-4121-b33f-a3880caf20ab response_code = 204 uri = /switch-ports
26292024-07-26T02:43:12.053ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 84 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:49998 req_id = ad2dea24-74ec-4243-a737-62ba7f128c0a response_code = 200 uri = /network-bootstore-config
26302024-07-26T02:43:16.813ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:54154
26312024-07-26T02:43:16.814ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 94 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:54154 req_id = 59574a84-b367-4ce1-935f-9161dab1d56f response_code = 200 uri = /baseboard
26322024-07-26T02:43:25.124ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 3189 local_addr = [fd00:1122:3344:101::1]:33355 method = GET remote_addr = [fd00:1122:3344:101::d]:44106 req_id = 96b1bded-fad6-49a4-bcc7-6541d11d3d22 response_code = 200 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83 uri = /ea0e0fc4-fc93-4de6-b687-98567d824e83
26332024-07-26T02:43:40.968ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:48687
26342024-07-26T02:43:40.969ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 114 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:48687 req_id = cdc801a6-e0b7-4eb9-b3ad-a98507889993 response_code = 200 uri = /vpc-routes
26352024-07-26T02:43:40.969ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:41189
26362024-07-26T02:43:40.970ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 1226 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:41189 req_id = 1dc21a5a-513d-44f3-82b0-50b2da821963 response_code = 200 uri = /v2p
26372024-07-26T02:43:40.976ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 127 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:48687 req_id = 4bc4c800-77b9-44f3-a4f8-f55df839f6a1 response_code = 204 uri = /vpc-routes
26382024-07-26T02:43:41.122ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:40715
26392024-07-26T02:43:41.122ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:673 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 2f46bf86-4fc6-46a8-b573-bf2fb626f25d, kind: Service { id: cf7587f8-33d7-4e89-bad7-ae8cdd556425 }, name: Name("external-dns-cf7587f8-33d7-4e89-bad7-ae8cdd556425"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 168, 111])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 567092fa-fb08-4940-8635-631e5c908557, kind: Service { id: 53979846-b429-4bae-9968-d96ff860d07e }, name: Name("external-dns-53979846-b429-4bae-9968-d96ff860d07e"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 158, 230])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 824ae30e-19ba-423f-923a-3afa2a59b03f, kind: Service { id: 8b4031b8-df71-45f8-b977-b6aa00214087 }, name: Name("nexus-8b4031b8-df71-45f8-b977-b6aa00214087"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 145])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 95c06611-4118-4e93-84ac-1b68f8ff2a6d, kind: Service { id: 9d80bc64-2ec6-4d4e-96d7-5c012d075087 }, name: Name("nexus-9d80bc64-2ec6-4d4e-96d7-5c012d075087"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 148])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e94e36b7-3a4e-4066-8a04-518f25a7ffdb, kind: Service { id: 56f1d4b9-214d-4104-bc80-a770b9be089e }, name: Name("nexus-56f1d4b9-214d-4104-bc80-a770b9be089e"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 180, 23])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
26402024-07-26T02:43:41.122ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
26412024-07-26T02:43:41.122ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
26422024-07-26T02:43:41.122ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26432024-07-26T02:43:41.122ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26442024-07-26T02:43:41.122ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte0" rules = []
26452024-07-26T02:43:41.122ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26462024-07-26T02:43:41.122ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 331 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:40715 req_id = 069f380f-c8bc-4ea3-a02e-2f40735f774f response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
26472024-07-26T02:43:41.148ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40220
26482024-07-26T02:43:41.148ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 29 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40220 req_id = 668cf6e1-63db-4468-8ef4-196e423cf474 response_code = 200 uri = /vpc-routes
26492024-07-26T02:43:41.155ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 64 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:40220 req_id = ccd4db0a-c9e4-49e8-afc1-c23a546e95a6 response_code = 204 uri = /vpc-routes
26502024-07-26T02:43:41.186ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:41762
26512024-07-26T02:43:41.187ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 100 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:41762 req_id = 3c7c3272-dec3-4327-97af-4bf8aaf7229b response_code = 200 uri = /v2p
26522024-07-26T02:43:41.311ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:33544
26532024-07-26T02:43:41.312ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
26542024-07-26T02:43:41.336ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
26552024-07-26T02:43:41.359ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 47906 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:33544 req_id = dc7e15cf-c1b6-49a1-872b-31c27263939b response_code = 204 uri = /switch-ports
26562024-07-26T02:43:41.360ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:59194
26572024-07-26T02:43:41.360ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:673 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 2f46bf86-4fc6-46a8-b573-bf2fb626f25d, kind: Service { id: cf7587f8-33d7-4e89-bad7-ae8cdd556425 }, name: Name("external-dns-cf7587f8-33d7-4e89-bad7-ae8cdd556425"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 168, 111])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 567092fa-fb08-4940-8635-631e5c908557, kind: Service { id: 53979846-b429-4bae-9968-d96ff860d07e }, name: Name("external-dns-53979846-b429-4bae-9968-d96ff860d07e"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 158, 230])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 824ae30e-19ba-423f-923a-3afa2a59b03f, kind: Service { id: 8b4031b8-df71-45f8-b977-b6aa00214087 }, name: Name("nexus-8b4031b8-df71-45f8-b977-b6aa00214087"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 145])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 95c06611-4118-4e93-84ac-1b68f8ff2a6d, kind: Service { id: 9d80bc64-2ec6-4d4e-96d7-5c012d075087 }, name: Name("nexus-9d80bc64-2ec6-4d4e-96d7-5c012d075087"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 148])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e94e36b7-3a4e-4066-8a04-518f25a7ffdb, kind: Service { id: 56f1d4b9-214d-4104-bc80-a770b9be089e }, name: Name("nexus-56f1d4b9-214d-4104-bc80-a770b9be089e"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 180, 23])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
26582024-07-26T02:43:41.360ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
26592024-07-26T02:43:41.360ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 265 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:33544 req_id = ca5ef92a-b61b-4f54-8577-33d71feabf8b response_code = 200 uri = /network-bootstore-config
26602024-07-26T02:43:41.360ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
26612024-07-26T02:43:41.360ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26622024-07-26T02:43:41.360ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26632024-07-26T02:43:41.360ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte0" rules = []
26642024-07-26T02:43:41.360ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26652024-07-26T02:43:41.360ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 492 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:59194 req_id = 32fc2e99-c69f-4f3e-a8a7-38e3626a47bc response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
26662024-07-26T02:43:41.436ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:43406
26672024-07-26T02:43:41.437ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
26682024-07-26T02:43:41.460ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
26692024-07-26T02:43:41.482ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 45073 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:43406 req_id = 5436d855-944d-4274-b0c4-94bfe01f3676 response_code = 204 uri = /switch-ports
26702024-07-26T02:43:41.482ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 92 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:43406 req_id = 8a498377-a1bc-49d7-a99b-642136f15312 response_code = 200 uri = /network-bootstore-config
26712024-07-26T02:43:41.850ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:37153
26722024-07-26T02:43:41.850ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 100 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:37153 req_id = d6f5a590-70ea-4cbe-b7a0-69aac3e66eb8 response_code = 200 uri = /vpc-routes
26732024-07-26T02:43:41.857ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 130 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:37153 req_id = d06cc4f3-3ee8-4892-9347-2776f3e0539b response_code = 204 uri = /vpc-routes
26742024-07-26T02:43:41.908ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:45439
26752024-07-26T02:43:41.908ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 194 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:45439 req_id = 3be8c615-5ac0-4ff2-b5ff-2dcae7a5624d response_code = 200 uri = /v2p
26762024-07-26T02:43:42.007ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36479
26772024-07-26T02:43:42.007ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:673 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 2f46bf86-4fc6-46a8-b573-bf2fb626f25d, kind: Service { id: cf7587f8-33d7-4e89-bad7-ae8cdd556425 }, name: Name("external-dns-cf7587f8-33d7-4e89-bad7-ae8cdd556425"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 168, 111])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 567092fa-fb08-4940-8635-631e5c908557, kind: Service { id: 53979846-b429-4bae-9968-d96ff860d07e }, name: Name("external-dns-53979846-b429-4bae-9968-d96ff860d07e"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 158, 230])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 824ae30e-19ba-423f-923a-3afa2a59b03f, kind: Service { id: 8b4031b8-df71-45f8-b977-b6aa00214087 }, name: Name("nexus-8b4031b8-df71-45f8-b977-b6aa00214087"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 161, 145])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 95c06611-4118-4e93-84ac-1b68f8ff2a6d, kind: Service { id: 9d80bc64-2ec6-4d4e-96d7-5c012d075087 }, name: Name("nexus-9d80bc64-2ec6-4d4e-96d7-5c012d075087"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 148])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e94e36b7-3a4e-4066-8a04-518f25a7ffdb, kind: Service { id: 56f1d4b9-214d-4104-bc80-a770b9be089e }, name: Name("nexus-56f1d4b9-214d-4104-bc80-a770b9be089e"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 180, 23])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
26782024-07-26T02:43:42.007ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
26792024-07-26T02:43:42.007ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
26802024-07-26T02:43:42.007ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26812024-07-26T02:43:42.007ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26822024-07-26T02:43:42.007ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte0" rules = []
26832024-07-26T02:43:42.007ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26842024-07-26T02:43:42.007ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 339 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:36479 req_id = 7a6d753f-b670-4d67-9f59-311297329cdb response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
26852024-07-26T02:43:42.362ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:35112
26862024-07-26T02:43:42.366ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
26872024-07-26T02:43:42.388ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
26882024-07-26T02:43:42.411ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 49051 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:35112 req_id = 3191f10c-29ca-40d2-8e9f-66be6e1e9f2c response_code = 204 uri = /switch-ports
26892024-07-26T02:43:42.412ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 884 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:35112 req_id = 0628f76c-a6ce-4c22-81b7-a776ce8c9c08 response_code = 200 uri = /network-bootstore-config
26902024-07-26T02:43:46.846ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:54286
26912024-07-26T02:43:46.846ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 58 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:54286 req_id = d3c05be1-f207-4047-9c28-9d8f0c19b2b3 response_code = 200 uri = /baseboard
26922024-07-26T02:43:55.123ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 2352 local_addr = [fd00:1122:3344:101::1]:33355 method = GET remote_addr = [fd00:1122:3344:101::d]:44106 req_id = a7c4d2bf-8800-49b7-a19f-42a4fffcd23b response_code = 200 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83 uri = /ea0e0fc4-fc93-4de6-b687-98567d824e83
26932024-07-26T02:43:55.850ZINFOSledAgent (auto-cleanup-task): running automatic periodic zone bundle cleanup file = sled-agent/src/zone_bundle.rs:303
26942024-07-26T02:43:55.851ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
26952024-07-26T02:43:55.890ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:292 next_cleanup = Instant { tv_sec: 13394, tv_nsec: 853998981 } time_to_next_cleanup = 599.9999979s
26962024-07-26T02:44:09.765ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
26972024-07-26T02:44:10.936ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:55493
26982024-07-26T02:44:10.936ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 188 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:55493 req_id = aafa3518-f113-4161-b763-d52a6f2100fb response_code = 200 uri = /v2p
26992024-07-26T02:44:10.995ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:53708
27002024-07-26T02:44:10.995ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 105 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:53708 req_id = 5f75a1e1-1b95-4075-bdd9-b1c6a1b1b7b5 response_code = 200 uri = /v2p
27012024-07-26T02:44:11.013ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:42559
27022024-07-26T02:44:11.013ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 32 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:42559 req_id = e57330f0-531b-4a49-b975-aabaf44b5b59 response_code = 200 uri = /vpc-routes
27032024-07-26T02:44:11.016ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:55310
27042024-07-26T02:44:11.016ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 22 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:55310 req_id = 3c1264e3-bfe2-4038-b876-aad2d27e0515 response_code = 200 uri = /vpc-routes
27052024-07-26T02:44:11.022ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 100 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:42559 req_id = 864f24a2-d560-44a9-9bf5-931f1ee7e969 response_code = 204 uri = /vpc-routes
27062024-07-26T02:44:11.023ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 74 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:55310 req_id = 1019cace-57ed-4f19-a1a5-6474949d5b3b response_code = 204 uri = /vpc-routes
27072024-07-26T02:44:11.103ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:41440
27082024-07-26T02:44:11.103ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
27092024-07-26T02:44:11.129ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
27102024-07-26T02:44:11.152ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 49473 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:41440 req_id = 0aeb01ff-49bc-48bd-b408-539c4bf310a9 response_code = 204 uri = /switch-ports
27112024-07-26T02:44:11.153ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:49093
27122024-07-26T02:44:11.153ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 278 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:41440 req_id = 39eac491-0c08-46cf-b988-9f64c645aabe response_code = 200 uri = /network-bootstore-config
27132024-07-26T02:44:11.154ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
27142024-07-26T02:44:11.179ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
27152024-07-26T02:44:11.201ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 47914 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:49093 req_id = c50724a0-1ccf-471f-b72b-84ee2b71e3bf response_code = 204 uri = /switch-ports
27162024-07-26T02:44:11.201ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 155 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:49093 req_id = be9daae2-842b-4870-b433-226c954a7d05 response_code = 200 uri = /network-bootstore-config
27172024-07-26T02:44:11.922ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:48232
27182024-07-26T02:44:11.922ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 511 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:48232 req_id = 27c4448f-9331-4584-8633-81202f1d5d80 response_code = 200 uri = /vpc-routes
27192024-07-26T02:44:11.927ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:55034
27202024-07-26T02:44:11.927ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 171 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:55034 req_id = ef8f7b49-a672-4fdc-b45f-1a9391a3e169 response_code = 200 uri = /v2p
27212024-07-26T02:44:11.933ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 128 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:48232 req_id = f6bdbf7d-efc9-4e31-9f77-ca02e4a516e9 response_code = 204 uri = /vpc-routes
27222024-07-26T02:44:12.003ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:60523
27232024-07-26T02:44:12.003ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
27242024-07-26T02:44:12.029ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
27252024-07-26T02:44:12.053ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 49833 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:60523 req_id = a0142b38-ebfc-4e8d-a2fc-93a05b8824bc response_code = 204 uri = /switch-ports
27262024-07-26T02:44:12.053ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:60523 req_id = 5db2350c-becc-44cf-99b7-2279e8d7ecb2 response_code = 200 uri = /network-bootstore-config
27272024-07-26T02:44:16.880ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:42778
27282024-07-26T02:44:16.880ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 99 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:42778 req_id = 6d04a316-9e41-4dfa-8b78-d1d58c164dc6 response_code = 200 uri = /baseboard
27292024-07-26T02:44:25.122ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 1557 local_addr = [fd00:1122:3344:101::1]:33355 method = GET remote_addr = [fd00:1122:3344:101::d]:44106 req_id = e329470f-192c-45e9-a375-4f4165eb7fcb response_code = 200 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83 uri = /ea0e0fc4-fc93-4de6-b687-98567d824e83
27302024-07-26T02:44:40.945ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:48521
27312024-07-26T02:44:40.945ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 108 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:48521 req_id = 0e825475-3bd8-426b-bfb4-57192ba8cd24 response_code = 200 uri = /vpc-routes
27322024-07-26T02:44:40.953ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 137 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:48521 req_id = 17671cd8-066f-4b36-b034-477d4f3e98f0 response_code = 204 uri = /vpc-routes
27332024-07-26T02:44:40.973ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36675
27342024-07-26T02:44:40.973ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 32 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:36675 req_id = a39c587f-d771-4d9d-9611-2f35bc2daca9 response_code = 200 uri = /vpc-routes
27352024-07-26T02:44:40.980ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 81 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:36675 req_id = afcbdc43-28ea-4d98-ab6c-1764b7c5e19e response_code = 204 uri = /vpc-routes
27362024-07-26T02:44:40.982ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40986
27372024-07-26T02:44:40.982ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40986 req_id = bce3d7db-a7a5-47ef-a118-d4fa0d4f3676 response_code = 200 uri = /v2p
27382024-07-26T02:44:41.100ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:64355
27392024-07-26T02:44:41.100ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 189 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:64355 req_id = a9960d91-1f75-4abe-8366-f61a22892038 response_code = 200 uri = /v2p
27402024-07-26T02:44:41.259ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:65149
27412024-07-26T02:44:41.259ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
27422024-07-26T02:44:41.284ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
27432024-07-26T02:44:41.309ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 50016 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:65149 req_id = 5cebf478-a3b8-464a-8036-4155d8a494d7 response_code = 204 uri = /switch-ports
27442024-07-26T02:44:41.309ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:34393
27452024-07-26T02:44:41.309ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
27462024-07-26T02:44:41.313ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 3719 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:65149 req_id = 72b98242-d659-42c5-8322-c755a43cb898 response_code = 200 uri = /network-bootstore-config
27472024-07-26T02:44:41.335ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
27482024-07-26T02:44:41.358ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 49362 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:34393 req_id = a15d2f91-d419-4a70-b265-29f50a99eec1 response_code = 204 uri = /switch-ports
27492024-07-26T02:44:41.359ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 151 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:34393 req_id = 4023b80a-1299-4ad8-bf57-a9d4c10de6b1 response_code = 200 uri = /network-bootstore-config
27502024-07-26T02:44:42.009ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61359
27512024-07-26T02:44:42.010ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 173 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:61359 req_id = a6c36d50-1ae0-43b8-a322-59bd9bcdec79 response_code = 200 uri = /v2p
27522024-07-26T02:44:42.130ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:55780
27532024-07-26T02:44:42.130ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 129 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:55780 req_id = e3ca3040-5779-4320-b447-4f5888d2f17a response_code = 200 uri = /vpc-routes
27542024-07-26T02:44:42.137ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 115 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:55780 req_id = aaaca2fa-447b-43bb-aaaf-03ed686a9883 response_code = 204 uri = /vpc-routes
27552024-07-26T02:44:42.253ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:38643
27562024-07-26T02:44:42.253ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
27572024-07-26T02:44:42.278ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
27582024-07-26T02:44:42.301ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 48591 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:38643 req_id = bed611fd-093e-4fc1-9746-bfb4cedcaf50 response_code = 204 uri = /switch-ports
27592024-07-26T02:44:42.301ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 89 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:38643 req_id = ab26bf28-32bd-45ac-91c4-eddd4fdaa225 response_code = 200 uri = /network-bootstore-config
27602024-07-26T02:44:46.914ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:58033
27612024-07-26T02:44:46.914ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 111 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:58033 req_id = c906c170-3204-4979-8779-0f34c1be29df response_code = 200 uri = /baseboard
27622024-07-26T02:44:55.121ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 598 local_addr = [fd00:1122:3344:101::1]:33355 method = GET remote_addr = [fd00:1122:3344:101::d]:44106 req_id = e68c9a8e-1ede-4024-8cb8-3de6fb494da9 response_code = 200 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83 uri = /ea0e0fc4-fc93-4de6-b687-98567d824e83
27632024-07-26T02:45:10.908ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:46706
27642024-07-26T02:45:10.908ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 96 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:46706 req_id = cd884401-43cd-461e-a6ad-61dd764219f2 response_code = 200 uri = /vpc-routes
27652024-07-26T02:45:10.910ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:47147
27662024-07-26T02:45:10.910ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 139 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:47147 req_id = b5f8f034-5850-446c-bc44-5af28638fb54 response_code = 200 uri = /v2p
27672024-07-26T02:45:10.913ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50056
27682024-07-26T02:45:10.914ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 89 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50056 req_id = 46a3b4af-f623-47cc-9b80-1b7581a14ccf response_code = 200 uri = /v2p
27692024-07-26T02:45:10.915ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 107 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:46706 req_id = 43c75a84-197f-43dd-9590-8dced295be95 response_code = 204 uri = /vpc-routes
27702024-07-26T02:45:11.005ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:33567
27712024-07-26T02:45:11.006ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 655 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:33567 req_id = 5ead8e98-ae1f-41d1-9fb8-ba09f7680b99 response_code = 200 uri = /vpc-routes
27722024-07-26T02:45:11.013ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 83 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:33567 req_id = 8126ab43-297f-4d20-8ebe-6f36a62d7dae response_code = 204 uri = /vpc-routes
27732024-07-26T02:45:11.118ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:41468
27742024-07-26T02:45:11.118ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
27752024-07-26T02:45:11.146ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
27762024-07-26T02:45:11.167ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 49452 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:41468 req_id = 0644b5e5-ce87-4576-9757-e14afaa78e86 response_code = 204 uri = /switch-ports
27772024-07-26T02:45:11.168ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 567 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:41468 req_id = 59dde202-1215-4270-9969-eea04e87e7e7 response_code = 200 uri = /network-bootstore-config
27782024-07-26T02:45:11.212ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36680
27792024-07-26T02:45:11.212ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
27802024-07-26T02:45:11.235ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
27812024-07-26T02:45:11.255ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 43276 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:36680 req_id = 39749632-01de-4216-8fd4-4ce72159e3e9 response_code = 204 uri = /switch-ports
27822024-07-26T02:45:11.256ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 700 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:36680 req_id = dc19a9ac-5f55-4bc2-bf59-573e566a42cf response_code = 200 uri = /network-bootstore-config
27832024-07-26T02:45:11.813ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41305
27842024-07-26T02:45:11.813ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 87 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41305 req_id = fededbeb-067d-4429-92bc-3338941798e9 response_code = 200 uri = /vpc-routes
27852024-07-26T02:45:11.819ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:43964
27862024-07-26T02:45:11.819ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 132 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:43964 req_id = 0f865705-303e-41eb-ad4b-fecaa939c743 response_code = 200 uri = /v2p
27872024-07-26T02:45:11.820ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 103 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:41305 req_id = 1ed3ccdc-613b-49b4-99cb-e2e3400aff1f response_code = 204 uri = /vpc-routes
27882024-07-26T02:45:12.026ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59197
27892024-07-26T02:45:12.026ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
27902024-07-26T02:45:12.051ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
27912024-07-26T02:45:12.074ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 48196 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:59197 req_id = 1d5c09aa-1c88-4109-8309-d0b32b1736dc response_code = 204 uri = /switch-ports
27922024-07-26T02:45:12.075ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 92 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:59197 req_id = faa03a72-c1e8-451e-a211-18a9abec5fd0 response_code = 200 uri = /network-bootstore-config
27932024-07-26T02:45:14.083ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:53590
27942024-07-26T02:45:14.084ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 754 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:53590 req_id = a0edf1d0-b1b2-4dac-9756-90d9981682db response_code = 200 uri = /v2p
27952024-07-26T02:45:14.084ZINFOSledAgent (PortManager): Mapping virtual NIC to physical host file = illumos-utils/src/opte/port_manager.rs:780 mapping = VirtualNetworkInterfaceHost { virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 254, 166, 16])), physical_host_ip: fd00:1122:3344:101::1, vni: Vni(9883005) }
27962024-07-26T02:45:14.084ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 612 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:53590 req_id = bf2ddbbe-625b-48b2-85b8-f1b7e723e92d response_code = 204 uri = /v2p
27972024-07-26T02:45:14.264ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:48728
27982024-07-26T02:45:14.311ZINFOSledAgent (InstanceManager): ensuring instance is registered file = sled-agent/src/instance_manager.rs:577 hardware = InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: 1c18881b-2d0d-40fc-b0ae-bff2888c2686, kind: Instance { id: 4339d003-d6c5-45d3-997b-9e16d27048e0 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 254, 166, 16])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(9883005), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.2.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.2.176), floating_ips: [], firewall_rules: [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 1c18881b-2d0d-40fc-b0ae-bff2888c2686, kind: Instance { id: 4339d003-d6c5-45d3-997b-9e16d27048e0 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 254, 166, 16])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(9883005), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 1c18881b-2d0d-40fc-b0ae-bff2888c2686, kind: Instance { id: 4339d003-d6c5-45d3-997b-9e16d27048e0 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 254, 166, 16])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(9883005), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(9883005))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 1c18881b-2d0d-40fc-b0ae-bff2888c2686, kind: Instance { id: 4339d003-d6c5-45d3-997b-9e16d27048e0 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 254, 166, 16])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(9883005), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-9a55edbf7aa", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 93202239-8905-4143-90c9-72c9c2c57851, read_only_parent: Some(Volume { block_size: 512, id: e987411a-338f-4162-9be3-96b3a0db18dc, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: a6916a24-7388-4413-b242-8924c309b31f, key: Some("R/SR4dIKzzlXrmeLKtmZK2t3475bq4aJM1yuNV7xByE="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::13]:19001", "[fd00:1122:3344:101::18]:19001", "[fd00:1122:3344:101::12]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 93202239-8905-4143-90c9-72c9c2c57851, key: Some("jmeSKXpRneOBscv+9emybfwWLqgljhgjMS1FaYBUuH4="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::12]:19002", "[fd00:1122:3344:101::14]:19001", "[fd00:1122:3344:101::16]:19000"] } }] } }], cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAAAVpxX6WPpYAACnFfpYAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAFqcV+lj6WAAApxX6WAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiI0MzM5ZDAwMy1kNmM1LTQ1ZDMtOTk3Yi05ZTE2ZDI3MDQ4ZTAiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJTVRkTE8zclQ3Uy90d2oydmY1Y3hvK3NlUTM1cG1nbXowV2loeU1QajJIZSJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA") } instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0 instance_runtime = InstanceRuntimeState { propolis_id: Some(400431bd-e96b-4ea7-9e2f-5a4d556819e4 (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-07-26T02:45:13.770859467Z } metadata = InstanceMetadata { silo_id: 2218f0bf-bef6-4785-b5b0-f8053a24df0b, project_id: c67ecc4c-8545-4ca9-94c2-44f13e076acb } propolis_addr = [fd00:1122:3344:101::1:0]:12400 propolis_id = 400431bd-e96b-4ea7-9e2f-5a4d556819e4 vmm_runtime = VmmRuntimeState { state: Starting, gen: Generation(1), time_updated: 2024-07-26T02:45:13.695276Z }
27992024-07-26T02:45:14.311ZINFOSledAgent (InstanceManager): registering new instance file = sled-agent/src/instance_manager.rs:611 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0 (instance)
28002024-07-26T02:45:14.311ZINFOSledAgent (InstanceManager): initializing new Instance file = sled-agent/src/instance.rs:951 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0 propolis_id = 400431bd-e96b-4ea7-9e2f-5a4d556819e4 state = InstanceInitialState { hardware: InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: 1c18881b-2d0d-40fc-b0ae-bff2888c2686, kind: Instance { id: 4339d003-d6c5-45d3-997b-9e16d27048e0 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 254, 166, 16])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(9883005), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.2.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.2.176), floating_ips: [], firewall_rules: [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 1c18881b-2d0d-40fc-b0ae-bff2888c2686, kind: Instance { id: 4339d003-d6c5-45d3-997b-9e16d27048e0 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 254, 166, 16])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(9883005), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 1c18881b-2d0d-40fc-b0ae-bff2888c2686, kind: Instance { id: 4339d003-d6c5-45d3-997b-9e16d27048e0 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 254, 166, 16])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(9883005), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(9883005))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 1c18881b-2d0d-40fc-b0ae-bff2888c2686, kind: Instance { id: 4339d003-d6c5-45d3-997b-9e16d27048e0 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 254, 166, 16])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(9883005), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-9a55edbf7aa", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 93202239-8905-4143-90c9-72c9c2c57851, read_only_parent: Some(Volume { block_size: 512, id: e987411a-338f-4162-9be3-96b3a0db18dc, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: a6916a24-7388-4413-b242-8924c309b31f, key: Some("R/SR4dIKzzlXrmeLKtmZK2t3475bq4aJM1yuNV7xByE="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::13]:19001", "[fd00:1122:3344:101::18]:19001", "[fd00:1122:3344:101::12]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 93202239-8905-4143-90c9-72c9c2c57851, key: Some("jmeSKXpRneOBscv+9emybfwWLqgljhgjMS1FaYBUuH4="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::12]:19002", "[fd00:1122:3344:101::14]:19001", "[fd00:1122:3344:101::16]:19000"] } }] } }], cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAAAVpxX6WPpYAACnFfpYAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAFqcV+lj6WAAApxX6WAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiI0MzM5ZDAwMy1kNmM1LTQ1ZDMtOTk3Yi05ZTE2ZDI3MDQ4ZTAiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJTVRkTE8zclQ3Uy90d2oydmY1Y3hvK3NlUTM1cG1nbXowV2loeU1QajJIZSJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA") }, instance_runtime: InstanceRuntimeState { propolis_id: Some(400431bd-e96b-4ea7-9e2f-5a4d556819e4 (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-07-26T02:45:13.770859467Z }, vmm_runtime: VmmRuntimeState { state: Starting, gen: Generation(1), time_updated: 2024-07-26T02:45:13.695276Z }, propolis_addr: [fd00:1122:3344:101::1:0]:12400 }
28012024-07-26T02:45:14.312ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 47595 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:48728 req_id = 6fef7c6e-77ac-49f0-9cd3-f787052d03b1 response_code = 200 uri = /instances/4339d003-d6c5-45d3-997b-9e16d27048e0
28022024-07-26T02:45:14.569ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:35205
28032024-07-26T02:45:14.574ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 4089 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:35205 req_id = 746f27f1-2068-4c9a-98c5-53a199250169 response_code = 200 uri = /instances/4339d003-d6c5-45d3-997b-9e16d27048e0/state
28042024-07-26T02:45:14.578ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:426 port = Port { inner: PortInner(PortData { name: "opte6", ip: 172.30.0.5, mac: MacAddr6([168, 64, 37, 254, 166, 16]), slot: 0, vni: Vni { inner: 9883005 }, subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.0.1 } }) }
28052024-07-26T02:45:14.578ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
28062024-07-26T02:45:14.612ZINFOSledAgent (InstanceManager): Configuring new Omicron zone: oxz_propolis-server_400431bd-e96b-4ea7-9e2f-5a4d556819e4 file = illumos-utils/src/zone.rs:316 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0
28072024-07-26T02:45:14.634ZINFOSledAgent (InstanceManager): Installing Omicron zone: oxz_propolis-server_400431bd-e96b-4ea7-9e2f-5a4d556819e4 file = illumos-utils/src/zone.rs:354 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0
28082024-07-26T02:45:16.328ZINFOSledAgent (InstanceManager): Profile for oxz_propolis-server_400431bd-e96b-4ea7-9e2f-5a4d556819e4: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="system/illumos/propolis-server"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlInstance0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1:0'/> <propval type="astring" name="listen_port" value='12400'/> <propval type="astring" name="metric_addr" value='[fd00:1122:3344:101::c]:12221'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0
28092024-07-26T02:45:16.328ZINFOSledAgent (InstanceManager): Booting oxz_propolis-server_400431bd-e96b-4ea7-9e2f-5a4d556819e4 zone file = illumos-utils/src/running_zone.rs:472 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0 zone = oxz_propolis-server_400431bd-e96b-4ea7-9e2f-5a4d556819e4
28102024-07-26T02:45:16.502ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_400431bd-e96b-4ea7-9e2f-5a4d556819e4") failed: Property not found. retry in 38.46868ms file = illumos-utils/src/svc.rs:36 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0 zone = oxz_propolis-server_400431bd-e96b-4ea7-9e2f-5a4d556819e4
28112024-07-26T02:45:16.549ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_400431bd-e96b-4ea7-9e2f-5a4d556819e4") failed: Property not found. retry in 68.56394ms file = illumos-utils/src/svc.rs:36 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0 zone = oxz_propolis-server_400431bd-e96b-4ea7-9e2f-5a4d556819e4
28122024-07-26T02:45:16.630ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_400431bd-e96b-4ea7-9e2f-5a4d556819e4") failed: Property not found. retry in 278.034185ms file = illumos-utils/src/svc.rs:36 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0 zone = oxz_propolis-server_400431bd-e96b-4ea7-9e2f-5a4d556819e4
28132024-07-26T02:45:16.925ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_400431bd-e96b-4ea7-9e2f-5a4d556819e4") failed: Property not found. retry in 342.866818ms file = illumos-utils/src/svc.rs:36 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0 zone = oxz_propolis-server_400431bd-e96b-4ea7-9e2f-5a4d556819e4
28142024-07-26T02:45:16.960ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:50127
28152024-07-26T02:45:16.960ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 823 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:50127 req_id = 40b447b4-e6ad-4d5d-9ae5-37c9be72dd8a response_code = 200 uri = /baseboard
28162024-07-26T02:45:17.282ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_400431bd-e96b-4ea7-9e2f-5a4d556819e4") failed: Property not found. retry in 683.412523ms file = illumos-utils/src/svc.rs:36 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0 zone = oxz_propolis-server_400431bd-e96b-4ea7-9e2f-5a4d556819e4
28172024-07-26T02:45:17.976ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_400431bd-e96b-4ea7-9e2f-5a4d556819e4") failed: Property not found. retry in 1.489075298s file = illumos-utils/src/svc.rs:36 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0 zone = oxz_propolis-server_400431bd-e96b-4ea7-9e2f-5a4d556819e4
28182024-07-26T02:45:19.475ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_400431bd-e96b-4ea7-9e2f-5a4d556819e4") failed: Property not found. retry in 765.708939ms file = illumos-utils/src/svc.rs:36 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0 zone = oxz_propolis-server_400431bd-e96b-4ea7-9e2f-5a4d556819e4
28192024-07-26T02:45:20.250ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_400431bd-e96b-4ea7-9e2f-5a4d556819e4") failed: Property not found. retry in 1.441382496s file = illumos-utils/src/svc.rs:36 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0 zone = oxz_propolis-server_400431bd-e96b-4ea7-9e2f-5a4d556819e4
28202024-07-26T02:45:21.702ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_400431bd-e96b-4ea7-9e2f-5a4d556819e4") failed: Property not found. retry in 1.098852606s file = illumos-utils/src/svc.rs:36 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0 zone = oxz_propolis-server_400431bd-e96b-4ea7-9e2f-5a4d556819e4
28212024-07-26T02:45:22.829ZINFOSledAgent (InstanceManager): Started propolis in zone: oxz_propolis-server_400431bd-e96b-4ea7-9e2f-5a4d556819e4 file = sled-agent/src/instance.rs:1460 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0
28222024-07-26T02:45:22.837ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_400431bd-e96b-4ea7-9e2f-5a4d556819e4") failed: Property not found. retry in 30.487007ms file = illumos-utils/src/svc.rs:36 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0
28232024-07-26T02:45:22.877ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_400431bd-e96b-4ea7-9e2f-5a4d556819e4") failed: Property not found. retry in 131.49786ms file = illumos-utils/src/svc.rs:36 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0
28242024-07-26T02:45:23.018ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_400431bd-e96b-4ea7-9e2f-5a4d556819e4") failed: Property not found. retry in 223.706481ms file = illumos-utils/src/svc.rs:36 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0
28252024-07-26T02:45:23.253ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_400431bd-e96b-4ea7-9e2f-5a4d556819e4") failed: Property not found. retry in 338.16019ms file = illumos-utils/src/svc.rs:36 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0
28262024-07-26T02:45:23.601ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_400431bd-e96b-4ea7-9e2f-5a4d556819e4") failed: Property not found. retry in 1.028576976s file = illumos-utils/src/svc.rs:36 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0
28272024-07-26T02:45:24.643ZINFOSledAgent (InstanceManager): Propolis SMF service is online file = sled-agent/src/instance.rs:1469 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0
28282024-07-26T02:45:24.680ZINFOSledAgent (InstanceManager): Propolis HTTP server online file = sled-agent/src/instance.rs:1483 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0
28292024-07-26T02:45:24.682ZINFOSledAgent (InstanceManager): Sending ensure request to propolis: InstanceEnsureRequest { cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAAAVpxX6WPpYAACnFfpYAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAFqcV+lj6WAAApxX6WAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiI0MzM5ZDAwMy1kNmM1LTQ1ZDMtOTk3Yi05ZTE2ZDI3MDQ4ZTAiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJTVRkTE8zclQ3Uy90d2oydmY1Y3hvK3NlUTM1cG1nbXowV2loeU1QajJIZSJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"), disks: [DiskRequest { device: "nvme", name: "disk-9a55edbf7aa", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 93202239-8905-4143-90c9-72c9c2c57851, read_only_parent: Some(Volume { block_size: 512, id: e987411a-338f-4162-9be3-96b3a0db18dc, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: a6916a24-7388-4413-b242-8924c309b31f, key: Some("R/SR4dIKzzlXrmeLKtmZK2t3475bq4aJM1yuNV7xByE="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::13]:19001", "[fd00:1122:3344:101::18]:19001", "[fd00:1122:3344:101::12]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 93202239-8905-4143-90c9-72c9c2c57851, key: Some("jmeSKXpRneOBscv+9emybfwWLqgljhgjMS1FaYBUuH4="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::12]:19002", "[fd00:1122:3344:101::14]:19001", "[fd00:1122:3344:101::16]:19000"] } }] } }], migrate: None, nics: [NetworkInterfaceRequest { name: "opte6", slot: Slot(0) }], properties: InstanceProperties { bootrom_id: 00000000-0000-0000-0000-000000000000, description: "Test description", id: 4339d003-d6c5-45d3-997b-9e16d27048e0, image_id: 00000000-0000-0000-0000-000000000000, memory: 1024, metadata: InstanceMetadata { project_id: c67ecc4c-8545-4ca9-94c2-44f13e076acb, silo_id: 2218f0bf-bef6-4785-b5b0-f8053a24df0b }, name: "localshark", vcpus: 2 } } file = sled-agent/src/instance.rs:698 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0
28302024-07-26T02:45:24.825ZINFOSledAgent (InstanceManager): result of instance_ensure call is Ok(InstanceEnsureResponse { migrate: None }) file = sled-agent/src/instance.rs:700 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0
28312024-07-26T02:45:24.830ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Starting), migration_status: NoMigration, time: 2024-07-26T02:45:24.829362576Z } file = sled-agent/src/instance.rs:584 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0
28322024-07-26T02:45:24.830ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:606 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0 new_instance_state = InstanceRuntimeState { propolis_id: Some(400431bd-e96b-4ea7-9e2f-5a4d556819e4 (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-07-26T02:45:13.770859467Z } new_vmm_state = VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-07-26T02:45:24.829362576Z } propolis_id = 400431bd-e96b-4ea7-9e2f-5a4d556819e4
28332024-07-26T02:45:24.830ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:503 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(400431bd-e96b-4ea7-9e2f-5a4d556819e4 (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-07-26T02:45:13.770859467Z }, propolis_id: 400431bd-e96b-4ea7-9e2f-5a4d556819e4 (propolis), vmm_state: VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-07-26T02:45:24.829362576Z }, migration_state: None }
28342024-07-26T02:45:24.924ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Running), migration_status: NoMigration, time: 2024-07-26T02:45:24.924643231Z } file = sled-agent/src/instance.rs:584 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0
28352024-07-26T02:45:24.924ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:606 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0 new_instance_state = InstanceRuntimeState { propolis_id: Some(400431bd-e96b-4ea7-9e2f-5a4d556819e4 (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-07-26T02:45:13.770859467Z } new_vmm_state = VmmRuntimeState { state: Running, gen: Generation(3), time_updated: 2024-07-26T02:45:24.924643231Z } propolis_id = 400431bd-e96b-4ea7-9e2f-5a4d556819e4
28362024-07-26T02:45:24.924ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:503 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(400431bd-e96b-4ea7-9e2f-5a4d556819e4 (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-07-26T02:45:13.770859467Z }, propolis_id: 400431bd-e96b-4ea7-9e2f-5a4d556819e4 (propolis), vmm_state: VmmRuntimeState { state: Running, gen: Generation(3), time_updated: 2024-07-26T02:45:24.924643231Z }, migration_state: None }
28372024-07-26T02:45:24.962ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42422
28382024-07-26T02:45:24.962ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 48 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:42422 req_id = ad8ab023-ec70-46eb-acd8-476bc6fe45bb response_code = 200 uri = /vpc-routes
28392024-07-26T02:45:25.013ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 210 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:42422 req_id = b9ee1c0e-c282-406d-ab30-d56de3dd36d9 response_code = 204 uri = /vpc-routes
28402024-07-26T02:45:25.057ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:43281
28412024-07-26T02:45:25.057ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 61 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:43281 req_id = c734e07d-515e-4567-aaa8-92858888b646 response_code = 200 uri = /vpc-routes
28422024-07-26T02:45:25.069ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 104 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:43281 req_id = 4d7e384a-f1af-4e95-aa44-06f7cacece05 response_code = 204 uri = /vpc-routes
28432024-07-26T02:45:25.203ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 736 local_addr = [fd00:1122:3344:101::1]:33355 method = GET remote_addr = [fd00:1122:3344:101::d]:44106 req_id = 816ff992-8d15-4c33-ad89-b1bc5cc0cb8e response_code = 200 sled_id = ea0e0fc4-fc93-4de6-b687-98567d824e83 uri = /ea0e0fc4-fc93-4de6-b687-98567d824e83
28442024-07-26T02:45:37.070ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54567
28452024-07-26T02:45:37.071ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 945 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:54567 req_id = 81fe7bc3-9ed4-45db-bd4c-76ff6ed38d84 response_code = 200 uri = /instances/4339d003-d6c5-45d3-997b-9e16d27048e0/state
28462024-07-26T02:45:37.071ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Stopping), migration_status: NoMigration, time: 2024-07-26T02:45:37.071330661Z } file = sled-agent/src/instance.rs:584 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0
28472024-07-26T02:45:37.071ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:606 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0 new_instance_state = InstanceRuntimeState { propolis_id: Some(400431bd-e96b-4ea7-9e2f-5a4d556819e4 (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-07-26T02:45:13.770859467Z } new_vmm_state = VmmRuntimeState { state: Stopping, gen: Generation(5), time_updated: 2024-07-26T02:45:37.071330661Z } propolis_id = 400431bd-e96b-4ea7-9e2f-5a4d556819e4
28482024-07-26T02:45:37.071ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:503 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(400431bd-e96b-4ea7-9e2f-5a4d556819e4 (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-07-26T02:45:13.770859467Z }, propolis_id: 400431bd-e96b-4ea7-9e2f-5a4d556819e4 (propolis), vmm_state: VmmRuntimeState { state: Stopping, gen: Generation(5), time_updated: 2024-07-26T02:45:37.071330661Z }, migration_state: None }
28492024-07-26T02:45:37.136ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Destroyed), migration_status: NoMigration, time: 2024-07-26T02:45:37.136468681Z } file = sled-agent/src/instance.rs:584 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0
28502024-07-26T02:45:37.136ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:606 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0 new_instance_state = InstanceRuntimeState { propolis_id: None, dst_propolis_id: None, migration_id: None, gen: Generation(5), time_updated: 2024-07-26T02:45:37.136468681Z } new_vmm_state = VmmRuntimeState { state: Destroyed, gen: Generation(6), time_updated: 2024-07-26T02:45:37.136468681Z } propolis_id = 400431bd-e96b-4ea7-9e2f-5a4d556819e4
28512024-07-26T02:45:37.136ZINFOSledAgent (InstanceManager): terminating VMM that has exited file = sled-agent/src/instance.rs:621 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0
28522024-07-26T02:45:37.136ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
28532024-07-26T02:45:37.137ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
28542024-07-26T02:45:37.137ZINFOSledAgent (ZoneBundler): creating zone bundle context = ZoneBundleContext { storage_dirs: ["/pool/int/2ea70c3f-a29d-4c51-ad99-4cf32a6e81b1/debug/bundle/zone", "/pool/int/f09c5312-7869-4743-8a0d-863eb3bf0156/debug/bundle/zone"], cause: TerminatedInstance, extra_log_dirs: ["/pool/ext/4b40b5f5-772a-4d5a-bbcd-d2fa50a0ded8/crypt/debug", "/pool/ext/b8ef9471-5b81-40ed-90b5-1103a4a9eb91/crypt/debug", "/pool/ext/a47af5d0-e565-4513-84ad-c553b7265a72/crypt/debug", "/pool/ext/2a33f7b1-2bfd-4239-9138-fb534f4ccb3f/crypt/debug", "/pool/ext/a38a1ea7-44a1-411f-bbb8-767c9a23135f/crypt/debug", "/pool/ext/cb102b97-0352-4c03-8d37-4c80c4475510/crypt/debug", "/pool/ext/c5566b16-9a9f-498b-a4b1-83ac43308db5/crypt/debug", "/pool/ext/136ed548-ab63-44da-a187-0fa13dc111a0/crypt/debug", "/pool/ext/3ed9ecd9-4aeb-4ec2-8876-b9c8ed7df016/crypt/debug"] } file = sled-agent/src/zone_bundle.rs:445 zone_name = oxz_propolis-server_400431bd-e96b-4ea7-9e2f-5a4d556819e4
28552024-07-26T02:45:37.175ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33729
28562024-07-26T02:45:37.176ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 706 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:33729 req_id = e0a246c7-df53-4162-a461-dc008dfe2796 response_code = 200 uri = /vpc-routes
28572024-07-26T02:45:37.189ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 138 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:33729 req_id = 100d6fd9-bb6a-4126-953a-f851d2d6071e response_code = 204 uri = /vpc-routes
28582024-07-26T02:45:37.788ZINFOSledAgent (ZoneBundler): finished zone bundle file = sled-agent/src/zone_bundle.rs:1246 metadata = ZoneBundleMetadata { id: ZoneBundleId { zone_name: "oxz_propolis-server_400431bd-e96b-4ea7-9e2f-5a4d556819e4", bundle_id: 9ea10160-1bed-4ef7-badb-c0082579c5e1 }, time_created: 2024-07-26T02:45:37.137860644Z, version: 0, cause: TerminatedInstance }
28592024-07-26T02:45:37.788ZWARNSledAgent (InstanceManager): Halting and removing zone: oxz_propolis-server_400431bd-e96b-4ea7-9e2f-5a4d556819e4 file = sled-agent/src/instance.rs:793 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0
28602024-07-26T02:45:39.541ZINFOSledAgent (InstanceManager): halt_and_remove_logged: Previous zone state: Running file = illumos-utils/src/zone.rs:270 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0
28612024-07-26T02:45:39.556ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:906 id = RouterId { vni: Vni(9883005), kind: System }
28622024-07-26T02:45:39.556ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:906 id = RouterId { vni: Vni(9883005), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }
28632024-07-26T02:45:39.556ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:503 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: None, dst_propolis_id: None, migration_id: None, gen: Generation(5), time_updated: 2024-07-26T02:45:37.136468681Z }, propolis_id: 400431bd-e96b-4ea7-9e2f-5a4d556819e4 (propolis), vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-07-26T02:45:39.556958193Z }, migration_state: None }
28642024-07-26T02:45:39.647ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41751
28652024-07-26T02:45:39.648ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 271 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41751 req_id = c1252adc-9db2-4eb9-ae6b-ae889f4fbf3e response_code = 200 uri = /v2p
28662024-07-26T02:45:39.648ZINFOSledAgent (PortManager): Mapping virtual NIC to physical host file = illumos-utils/src/opte/port_manager.rs:780 mapping = VirtualNetworkInterfaceHost { virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 254, 166, 16])), physical_host_ip: fd00:1122:3344:101::1, vni: Vni(9883005) }
28672024-07-26T02:45:39.648ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:41751 req_id = 40f3ee68-f360-4992-a62c-a7a460a05c1a response_code = 204 uri = /v2p
28682024-07-26T02:45:39.829ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:503 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: None, dst_propolis_id: None, migration_id: None, gen: Generation(5), time_updated: 2024-07-26T02:45:37.136468681Z }, propolis_id: 400431bd-e96b-4ea7-9e2f-5a4d556819e4 (propolis), vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-07-26T02:45:39.556958193Z }, migration_state: None }
28692024-07-26T02:45:39.832ZINFOSledAgent (InstanceManager): State monitoring task complete file = sled-agent/src/instance.rs:736 instance_id = 4339d003-d6c5-45d3-997b-9e16d27048e0
28702024-07-26T02:45:39.872ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41172
28712024-07-26T02:45:39.873ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 717 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41172 req_id = cb0848c2-0f93-48c5-b4be-5a82ecc53be7 response_code = 200 uri = /vpc-routes
28722024-07-26T02:45:39.879ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 106 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:41172 req_id = 9efdf556-1e24-44ea-8609-88c90cb929d3 response_code = 204 uri = /vpc-routes
28732024-07-26T02:45:39.917ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:37300
28742024-07-26T02:45:39.918ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 42 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:37300 req_id = a476f4c0-4142-4bdd-95cc-a96601dfbff3 response_code = 200 uri = /vpc-routes
28752024-07-26T02:45:39.925ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 613 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:37300 req_id = ee90ee6e-546a-49da-a5d1-a5203a29661d response_code = 204 uri = /vpc-routes
28762024-07-26T02:45:40.082ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47924
28772024-07-26T02:45:40.082ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 136 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47924 req_id = bfa70343-7bfa-4352-b40d-1ee1c3210ae4 response_code = 200 uri = /v2p
28782024-07-26T02:45:40.082ZINFOSledAgent (PortManager): Clearing mapping of virtual NIC to physical host file = illumos-utils/src/opte/port_manager.rs:820 mapping = VirtualNetworkInterfaceHost { virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 254, 166, 16])), physical_host_ip: fd00:1122:3344:101::1, vni: Vni(9883005) }
28792024-07-26T02:45:40.083ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 91 local_addr = [fd00:1122:3344:101::1]:12345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:47924 req_id = 8de5140a-ac4d-4cc3-9fb5-342f2fee2ad8 response_code = 204 uri = /v2p
28802024-07-26T02:45:40.949ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:65393
28812024-07-26T02:45:40.949ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 89 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:65393 req_id = f8f3a9ef-d01f-40d2-82ad-955914f024c2 response_code = 200 uri = /vpc-routes
28822024-07-26T02:45:40.958ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 1167 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:65393 req_id = a13c8226-d2f7-463e-8fa6-295d70b69168 response_code = 204 uri = /vpc-routes
28832024-07-26T02:45:41.004ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:45288
28842024-07-26T02:45:41.005ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 1081 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:45288 req_id = 9fbe8732-9e54-4175-ba53-2c7507e3a952 response_code = 200 uri = /v2p
28852024-07-26T02:45:41.136ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:38723
28862024-07-26T02:45:41.136ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 37 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:38723 req_id = cab047c8-8c5d-4b70-8e89-490d44374959 response_code = 200 uri = /vpc-routes
28872024-07-26T02:45:41.142ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 95 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:38723 req_id = 1e5e85f4-0d44-4522-9439-1673dcd8f04b response_code = 204 uri = /vpc-routes
28882024-07-26T02:45:41.145ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:39735
28892024-07-26T02:45:41.145ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 22 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:39735 req_id = ca04193f-10ec-4c47-a3a0-ca7c994fc03a response_code = 200 uri = /vpc-routes
28902024-07-26T02:45:41.152ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 67 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:39735 req_id = e8a5e042-d00d-4bd9-9253-0ba08444cbce response_code = 204 uri = /vpc-routes
28912024-07-26T02:45:41.178ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50535
28922024-07-26T02:45:41.178ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
28932024-07-26T02:45:41.205ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
28942024-07-26T02:45:41.228ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 50116 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:50535 req_id = ca527727-3fd9-4264-9fcb-a2fea8543f7f response_code = 204 uri = /switch-ports
28952024-07-26T02:45:41.228ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:52717
28962024-07-26T02:45:41.229ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 760 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:52717 req_id = 2856ebd0-bba2-4a02-9522-1cf8a1a9e706 response_code = 200 uri = /v2p
28972024-07-26T02:45:41.229ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 721 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50535 req_id = 32034632-52ac-49ac-819c-54213decea09 response_code = 200 uri = /network-bootstore-config
28982024-07-26T02:45:41.325ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:42530
28992024-07-26T02:45:41.325ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
29002024-07-26T02:45:41.350ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
29012024-07-26T02:45:41.372ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 47104 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:42530 req_id = 661a9133-4779-4c73-8fcd-22f9e16c87f3 response_code = 204 uri = /switch-ports
29022024-07-26T02:45:41.372ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 103 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:42530 req_id = f6e3c3fa-ebae-43bb-a418-79cfcb9bf17f response_code = 200 uri = /network-bootstore-config
29032024-07-26T02:45:41.882ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:38263
29042024-07-26T02:45:41.882ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 166 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:38263 req_id = 194edc87-089b-474c-a5d2-08d5ed6dad69 response_code = 200 uri = /v2p
29052024-07-26T02:45:42.010ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:64060
29062024-07-26T02:45:42.011ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 58 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:64060 req_id = ddccc7e1-b90e-4906-957e-b861f49c1e7d response_code = 200 uri = /vpc-routes
29072024-07-26T02:45:42.019ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 110 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:64060 req_id = afb160be-f7a5-432e-9f38-4092c23ff430 response_code = 204 uri = /vpc-routes
29082024-07-26T02:45:42.271ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:65496
29092024-07-26T02:45:42.271ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3820
29102024-07-26T02:45:42.295ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3831
29112024-07-26T02:45:42.316ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 45753 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:65496 req_id = 1b63c384-1cdf-4fff-80a3-84ed52e76f7a response_code = 204 uri = /switch-ports
29122024-07-26T02:45:42.317ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/7b594d0/dropshot/src/server.rs:914 latency_us = 90 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:65496 req_id = 888eab76-6107-4697-9f48-6978fb48bb3d response_code = 200 uri = /network-bootstore-config