Max level shown:
1 [ May 15 07:08:05 Enabled. ]
2 [ May 15 07:08:05 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]
3 [ May 15 07:08:05 Method "start" exited with status 0. ]
4 [ May 15 07:08:05 Rereading configuration. ]
5 [ May 15 07:08:05 No 'refresh' method defined. Treating as :true. ]
6 note: configured to log to "/dev/stdout"
72024-05-15T07:08:05.477ZINFOSledAgent: Ensuring contract reaper thread file = illumos-utils/src/running_zone.rs:163
82024-05-15T07:08:05.477ZINFOSledAgent (BootstrapAgentStartup): finding links ["net0", "net1"] file = sled-agent/src/bootstrap/pre_server.rs:255
92024-05-15T07:08:05.530ZINFOSledAgent (BootstrapAgentStartup): Starting mg-ddm service addr-objs = [AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }] file = sled-agent/src/bootstrap/pre_server.rs:262
102024-05-15T07:08:05.530ZINFOSledAgent (BootstrapAgentStartup): Importing mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:42 path = /opt/oxide/mg-ddm/pkg/ddm/manifest.xml
112024-05-15T07:08:05.641ZINFOSledAgent (BootstrapAgentStartup): Setting mg-ddm interfaces file = sled-agent/src/bootstrap/maghemite.rs:50 interfaces = ("net0/ll" "net1/ll")
122024-05-15T07:08:05.650ZINFOSledAgent (BootstrapAgentStartup): Enabling mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:56
132024-05-15T07:08:05.658ZINFOSledAgent (BootstrapAgentStartup): Ensuring zfs key directory exists file = sled-agent/src/bootstrap/pre_server.rs:272 path = /var/run/oxide/
142024-05-15T07:08:06.258ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:87 prefix = Ipv6Prefix { addr: fdb0:18c0:4d0d:a02a::, len: 64 }
152024-05-15T07:08:06.397ZINFOSledAgent: Starting KeyManager file = sled-agent/src/long_running_tasks.rs:119
162024-05-15T07:08:06.397ZINFOSledAgent: Starting StorageManager file = sled-agent/src/long_running_tasks.rs:131
172024-05-15T07:08:06.397ZINFOSledAgent: Starting StorageMonitor file = sled-agent/src/long_running_tasks.rs:141
182024-05-15T07:08:06.397ZINFOSledAgent: Starting HardwareManager file = sled-agent/src/long_running_tasks.rs:162 nongimlet_observed_disks = [] sled_mode = Scrimlet { asic: SoftNpuZone }
192024-05-15T07:08:06.397ZINFOSledAgent (DumpSetup-worker): DumpSetup poll loop started. file = sled-agent/src/dump_setup.rs:506
202024-05-15T07:08:06.397ZINFOSledAgent (HardwareManager): Creating HardwareManager file = sled-hardware/src/illumos/mod.rs:625
212024-05-15T07:08:06.400ZINFOSledAgent (HardwareManager): Generated i86pc baseboard Pc { identifier: "sock", model: "i86pc" } file = sled-hardware/src/illumos/mod.rs:535
222024-05-15T07:08:06.400ZWARNSledAgent (HardwareManager): Device is not a Gimlet (i86pc), proceeding with null hardware view file = sled-hardware/src/illumos/mod.rs:674
232024-05-15T07:08:06.400ZINFOSledAgent: Starting HardwareMonitor file = sled-agent/src/long_running_tasks.rs:176
242024-05-15T07:08:06.400ZINFOSledAgent (HardwareManager): Monitoring for hardware updates file = sled-hardware/src/illumos/mod.rs:733
252024-05-15T07:08:06.400ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/m2_0.vdev
262024-05-15T07:08:06.400ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:262 underlay_network_info = None
272024-05-15T07:08:06.401ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }), tx: ... } file = sled-storage/src/manager.rs:381
282024-05-15T07:08:06.401ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }
292024-05-15T07:08:06.401ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:112 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" } path = /scratch/m2_0.vdev
302024-05-15T07:08:06.413ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_0.vdev file = sled-hardware/src/disk.rs:304
312024-05-15T07:08:06.413ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:315 id = c3ed08c3-e0e1-43f2-b040-3c4755a2167c (zpool)
322024-05-15T07:08:06.514ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: c3ed08c3-e0e1-43f2-b040-3c4755a2167c (zpool), kind: Internal }
332024-05-15T07:08:06.714ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:381
342024-05-15T07:08:06.714ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/m2_1.vdev
352024-05-15T07:08:06.714ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }
362024-05-15T07:08:06.714ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }
372024-05-15T07:08:06.714ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }), tx: ... } file = sled-storage/src/manager.rs:381
382024-05-15T07:08:06.714ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }
392024-05-15T07:08:06.714ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:112 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" } path = /scratch/m2_1.vdev
402024-05-15T07:08:06.722ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_1.vdev file = sled-hardware/src/disk.rs:304
412024-05-15T07:08:06.722ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:315 id = 3907cd6d-3c6d-4543-92de-9ec01f9bc32c (zpool)
422024-05-15T07:08:06.806ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 3907cd6d-3c6d-4543-92de-9ec01f9bc32c (zpool), kind: Internal }
432024-05-15T07:08:07.003ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: c3ed08c3-e0e1-43f2-b040-3c4755a2167c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 3907cd6d-3c6d-4543-92de-9ec01f9bc32c (zpool), kind: Internal } }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
442024-05-15T07:08:07.003ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_0.vdev
452024-05-15T07:08:07.003ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
462024-05-15T07:08:07.003ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
472024-05-15T07:08:07.003ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 }), tx: ... } file = sled-storage/src/manager.rs:381
482024-05-15T07:08:07.003ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }
492024-05-15T07:08:07.003ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_1.vdev
502024-05-15T07:08:07.003ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: c3ed08c3-e0e1-43f2-b040-3c4755a2167c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 3907cd6d-3c6d-4543-92de-9ec01f9bc32c (zpool), kind: Internal } }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
512024-05-15T07:08:07.003ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
522024-05-15T07:08:07.003ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
532024-05-15T07:08:07.003ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 }), tx: ... } file = sled-storage/src/manager.rs:381
542024-05-15T07:08:07.003ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }
552024-05-15T07:08:07.003ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_2.vdev
562024-05-15T07:08:07.003ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: c3ed08c3-e0e1-43f2-b040-3c4755a2167c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 3907cd6d-3c6d-4543-92de-9ec01f9bc32c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
572024-05-15T07:08:07.003ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
582024-05-15T07:08:07.003ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
592024-05-15T07:08:07.003ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 }), tx: ... } file = sled-storage/src/manager.rs:381
602024-05-15T07:08:07.003ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }
612024-05-15T07:08:07.004ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_3.vdev
622024-05-15T07:08:07.004ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: c3ed08c3-e0e1-43f2-b040-3c4755a2167c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 3907cd6d-3c6d-4543-92de-9ec01f9bc32c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
632024-05-15T07:08:07.004ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 }), tx: ... } file = sled-storage/src/manager.rs:381
642024-05-15T07:08:07.004ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }
652024-05-15T07:08:07.004ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
662024-05-15T07:08:07.004ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
672024-05-15T07:08:07.004ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_4.vdev
682024-05-15T07:08:07.004ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: c3ed08c3-e0e1-43f2-b040-3c4755a2167c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 3907cd6d-3c6d-4543-92de-9ec01f9bc32c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
692024-05-15T07:08:07.004ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, variant: U2, slot: 1030 }), tx: ... } file = sled-storage/src/manager.rs:381
702024-05-15T07:08:07.004ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
712024-05-15T07:08:07.004ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }
722024-05-15T07:08:07.004ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
732024-05-15T07:08:07.004ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_5.vdev
742024-05-15T07:08:07.004ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, variant: U2, slot: 1031 }), tx: ... } file = sled-storage/src/manager.rs:381
752024-05-15T07:08:07.004ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: c3ed08c3-e0e1-43f2-b040-3c4755a2167c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 3907cd6d-3c6d-4543-92de-9ec01f9bc32c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, variant: U2, slot: 1030 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
762024-05-15T07:08:07.004ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }
772024-05-15T07:08:07.004ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
782024-05-15T07:08:07.004ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
792024-05-15T07:08:07.004ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_6.vdev
802024-05-15T07:08:07.004ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: c3ed08c3-e0e1-43f2-b040-3c4755a2167c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 3907cd6d-3c6d-4543-92de-9ec01f9bc32c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, variant: U2, slot: 1030 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, variant: U2, slot: 1031 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
812024-05-15T07:08:07.004ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }, variant: U2, slot: 1032 }), tx: ... } file = sled-storage/src/manager.rs:381
822024-05-15T07:08:07.004ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
832024-05-15T07:08:07.004ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }
842024-05-15T07:08:07.004ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
852024-05-15T07:08:07.004ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_7.vdev
862024-05-15T07:08:07.004ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: c3ed08c3-e0e1-43f2-b040-3c4755a2167c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 3907cd6d-3c6d-4543-92de-9ec01f9bc32c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, variant: U2, slot: 1030 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, variant: U2, slot: 1031 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }, variant: U2, slot: 1032 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
872024-05-15T07:08:07.004ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }, variant: U2, slot: 1033 }), tx: ... } file = sled-storage/src/manager.rs:381
882024-05-15T07:08:07.004ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
892024-05-15T07:08:07.004ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
902024-05-15T07:08:07.004ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }
912024-05-15T07:08:07.004ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_8.vdev
922024-05-15T07:08:07.004ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }, variant: U2, slot: 1034 }), tx: ... } file = sled-storage/src/manager.rs:381
932024-05-15T07:08:07.004ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }
942024-05-15T07:08:07.004ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: c3ed08c3-e0e1-43f2-b040-3c4755a2167c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 3907cd6d-3c6d-4543-92de-9ec01f9bc32c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, variant: U2, slot: 1030 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, variant: U2, slot: 1031 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }, variant: U2, slot: 1032 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }, variant: U2, slot: 1033 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
952024-05-15T07:08:07.004ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
962024-05-15T07:08:07.004ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
972024-05-15T07:08:07.004ZINFOSledAgent: Waiting for boot disk file = sled-agent/src/long_running_tasks.rs:91
982024-05-15T07:08:07.004ZINFOSledAgent: Found boot disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" } file = sled-agent/src/long_running_tasks.rs:93
992024-05-15T07:08:07.004ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: c3ed08c3-e0e1-43f2-b040-3c4755a2167c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 3907cd6d-3c6d-4543-92de-9ec01f9bc32c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, variant: U2, slot: 1030 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, variant: U2, slot: 1031 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }, variant: U2, slot: 1032 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }, variant: U2, slot: 1033 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }, variant: U2, slot: 1034 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
1002024-05-15T07:08:07.004ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
1012024-05-15T07:08:07.004ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
1022024-05-15T07:08:07.004ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1032024-05-15T07:08:07.004ZINFOSledAgent: Starting Bootstore file = sled-agent/src/long_running_tasks.rs:200
1042024-05-15T07:08:07.004ZINFOSledAgent (bootstore): No ledger in /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1052024-05-15T07:08:07.004ZINFOSledAgent (bootstore): No ledger in /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1062024-05-15T07:08:07.004ZINFOSledAgent (bootstore): No ledger found. Loading Fsm::State as Uninitialized file = bootstore/src/schemes/v0/storage.rs:83 peer_id = pc-sock-i86pc
1072024-05-15T07:08:07.004ZINFOSledAgent (bootstore): No ledger in /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1082024-05-15T07:08:07.004ZINFOSledAgent (bootstore): No ledger in /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1092024-05-15T07:08:07.004ZINFOSledAgent (bootstore): No ledger found for network config file = bootstore/src/schemes/v0/storage.rs:146 peer_id = pc-sock-i86pc
1102024-05-15T07:08:07.004ZINFOSledAgent: Starting Bootstore DDMD poller file = sled-agent/src/long_running_tasks.rs:205
1112024-05-15T07:08:07.004ZINFOSledAgent: Starting ZoneBundler related tasks file = sled-agent/src/long_running_tasks.rs:220
1122024-05-15T07:08:07.015ZINFOSledAgent (ServiceManager): Creating ServiceManager file = sled-agent/src/services.rs:711
1132024-05-15T07:08:07.015ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:289 next_cleanup = Instant { tv_sec: 1251, tv_nsec: 643535330 } time_to_next_cleanup = 599.999886978s
1142024-05-15T07:08:07.015ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3617
1152024-05-15T07:08:07.015ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1162024-05-15T07:08:07.015ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/sled-agent-request.json file = common/src/ledger.rs:188
1172024-05-15T07:08:07.015ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/sled-agent-request.json file = common/src/ledger.rs:188
1182024-05-15T07:08:07.018ZINFOSledAgent (dropshot (BootstrapAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:196 local_addr = [fdb0:18c0:4d0d:a02a::1]:80
1192024-05-15T07:08:07.018ZINFOSledAgent (SledAgentSprocketsServer): Started listening file = sled-agent/src/bootstrap/sprockets_server.rs:44 local_addr = [fdb0:18c0:4d0d:a02a::1]:12346
1202024-05-15T07:08:07.019ZINFOSledAgent (RSS): Injecting RSS configuration: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.2.164, last: 10.151.2.173, }, ), ], external_dns_ips: [ 10.151.2.164, 10.151.2.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV1 { rack_subnet: Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56, }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV1 { routes: [ RouteConfig { destination: V4( Ipv4Network { addr: 0.0.0.0, prefix: 0, }, ), nexthop: 10.151.2.100, vlan_id: None, }, ], addresses: [ V4( Ipv4Network { addr: 10.151.2.174, prefix: 24, }, ), ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, } file = sled-agent/src/rack_setup/service.rs:1003
1212024-05-15T07:08:07.019ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
1222024-05-15T07:08:07.019ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1232024-05-15T07:08:07.019ZINFOSledAgent (RSS): No ledger in /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1242024-05-15T07:08:07.019ZINFOSledAgent (RSS): No ledger in /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1252024-05-15T07:08:07.019ZINFOSledAgent (RSS): RSS configuration has not been fully applied yet file = sled-agent/src/rack_setup/service.rs:1063
1262024-05-15T07:08:07.019ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1272024-05-15T07:08:07.019ZINFOSledAgent (RSS): No ledger in /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/rss-sled-plan.json file = common/src/ledger.rs:188
1282024-05-15T07:08:07.019ZINFOSledAgent (RSS): No ledger in /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/rss-sled-plan.json file = common/src/ledger.rs:188
1292024-05-15T07:08:07.019ZINFOSledAgent (RSS): Creating new allocation plan file = sled-agent/src/rack_setup/service.rs:1104
1302024-05-15T07:08:07.019ZINFOSledAgent (RSS): Serializing plan file = sled-agent/src/rack_setup/plan/sled.rs:118
1312024-05-15T07:08:07.019ZINFOSledAgent (RSS): Creating plan for the sled at fdb0:18c0:4d0d:a02a::1 file = sled-agent/src/rack_setup/plan/sled.rs:91
1322024-05-15T07:08:07.019ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1332024-05-15T07:08:07.019ZINFOSledAgent (RSS): Writing ledger to /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1342024-05-15T07:08:07.019ZINFOSledAgent (RSS): Writing ledger to /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1352024-05-15T07:08:07.019ZINFOSledAgent (RSS): Sled plan written to storage file = sled-agent/src/rack_setup/plan/sled.rs:138
1362024-05-15T07:08:07.019ZINFOSledAgent (RSS): Writing Rack Network Configuration to bootstore file = sled-agent/src/rack_setup/service.rs:1137
1372024-05-15T07:08:07.020ZINFOSledAgent (bootstore): Attempting to update network config with generation: 1, current_generation: 0 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-sock-i86pc
1382024-05-15T07:08:07.020ZINFOSledAgent (bootstore): Writing ledger to /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
1392024-05-15T07:08:07.020ZINFOSledAgent (bootstore): Writing ledger to /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
1402024-05-15T07:08:07.020ZINFOSledAgent (bootstore): Broadcasting network config with generation 1 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-sock-i86pc
1412024-05-15T07:08:07.020ZINFOSledAgent (BootstrapAgentRssHandler): Received initialization request from RSS file = sled-agent/src/bootstrap/rss_handle.rs:217 request = StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: acb2a796-8e9f-4811-b601-726a12d79b0b, rack_id: 89a4c48a-3dff-478b-98cb-9f441a913804, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net(Ipv6Network { addr: fd00:1122:3344:101::, prefix: 64 }) } } } target_sled = [fdb0:18c0:4d0d:a02a::1]:12346
1422024-05-15T07:08:07.020ZINFOSledAgent (SledAgentSprocketsServer): Accepted connection file = sled-agent/src/bootstrap/sprockets_server.rs:66 remote_addr = [fdb0:18c0:4d0d:a02a::1]:53471
1432024-05-15T07:08:07.020ZINFOSledAgent (SledAgentMain): Loading Sled Agent: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: acb2a796-8e9f-4811-b601-726a12d79b0b, rack_id: 89a4c48a-3dff-478b-98cb-9f441a913804, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net(Ipv6Network { addr: fd00:1122:3344:101::, prefix: 64 }) } } } file = sled-agent/src/bootstrap/server.rs:360
1442024-05-15T07:08:07.020ZINFOSledAgent (SledAgentMain): KeyManager: using hardcoded secret retriever file = sled-agent/src/bootstrap/server.rs:375
1452024-05-15T07:08:07.020ZINFOSledAgent (StorageManager): Received KeyManagerReady file = sled-storage/src/manager.rs:381
1462024-05-15T07:08:07.020ZINFOSledAgent: setting up sled agent server file = sled-agent/src/server.rs:44
1472024-05-15T07:08:07.020ZINFOSledAgent (StorageManager): Loading storage config file = sled-storage/src/manager.rs:530
1482024-05-15T07:08:07.020ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
1492024-05-15T07:08:07.020ZINFOSledAgent (StorageManager): No ledger in /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_ledger
1502024-05-15T07:08:07.020ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:87 prefix = Ipv6Prefix { addr: fd00:1122:3344:101::, len: 64 }
1512024-05-15T07:08:07.020ZINFOSledAgent (StorageManager): No ledger in /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_ledger
1522024-05-15T07:08:07.020ZINFOSledAgent (StorageManager): No ledger of physical disks exists file = sled-storage/src/manager.rs:519
1532024-05-15T07:08:07.020ZINFOSledAgent (StorageManager): KeyManager ready, but no ledger detected file = sled-storage/src/manager.rs:547
1542024-05-15T07:08:07.023ZINFOSledAgent (ServiceManager): Enabling switch zone (new) file = sled-agent/src/services.rs:3849
1552024-05-15T07:08:07.030ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }
1562024-05-15T07:08:07.041ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }
1572024-05-15T07:08:07.051ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }
1582024-05-15T07:08:07.060ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }
1592024-05-15T07:08:07.069ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }
1602024-05-15T07:08:07.077ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }
1612024-05-15T07:08:07.086ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }
1622024-05-15T07:08:07.094ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }
1632024-05-15T07:08:07.102ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }
1642024-05-15T07:08:07.102ZINFOSledAgent (StorageManager): No disks to be automatically managed file = sled-storage/src/manager.rs:558
1652024-05-15T07:08:07.102ZINFOSledAgent (StorageManager): automatically managing disks file = sled-storage/src/manager.rs:370
1662024-05-15T07:08:07.102ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:339
1672024-05-15T07:08:07.102ZINFOSledAgent (StorageManager): Successfully synchronized disks without error file = sled-storage/src/manager.rs:449 result = DisksManagementResult { status: [] }
1682024-05-15T07:08:07.102ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1692024-05-15T07:08:07.114ZINFOSledAgent: SledAgent::new(..) starting file = sled-agent/src/sled_agent.rs:355 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
1702024-05-15T07:08:07.114ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1712024-05-15T07:08:07.114ZINFOSledAgent: Requested swap device of size 64 GiB file = sled-agent/src/sled_agent.rs:367 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
1722024-05-15T07:08:07.114ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1732024-05-15T07:08:07.120ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 79.423477ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "701c516a-e11d-4925-a578-f7ae5782a9d5", "content-length": "124", "date": "Wed, 15 May 2024 07:08:06 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "701c516a-e11d-4925-a578-f7ae5782a9d5" } file = clients/ddm-admin-client/src/lib.rs:178
1742024-05-15T07:08:07.124ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:315
1752024-05-15T07:08:07.127ZINFOSledAgent: attempting to create encrypted zvol: name="oxi_c3ed08c3-e0e1-43f2-b040-3c4755a2167c/swap", size_gb=64 file = sled-agent/src/swap_device.rs:153
1762024-05-15T07:08:07.142ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:354
1772024-05-15T07:08:07.150ZINFOSledAgent: successfully created encrypted zvol: name="oxi_c3ed08c3-e0e1-43f2-b040-3c4755a2167c/swap", size_gb=64 file = sled-agent/src/swap_device.rs:249
1782024-05-15T07:08:07.150ZINFOSledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_c3ed08c3-e0e1-43f2-b040-3c4755a2167c/swap" file = sled-agent/src/swap_device.rs:96
1792024-05-15T07:08:07.155ZINFOSledAgent: Mounting backing filesystems file = sled-agent/src/sled_agent.rs:382 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
1802024-05-15T07:08:07.155ZINFOSledAgent (BackingFs): Processing fmd file = sled-agent/src/backing_fs.rs:126
1812024-05-15T07:08:07.155ZINFOSledAgent (BackingFs): Ensuring dataset oxi_c3ed08c3-e0e1-43f2-b040-3c4755a2167c/backing/fmd file = sled-agent/src/backing_fs.rs:136
1822024-05-15T07:08:07.200ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1832024-05-15T07:08:07.201ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 121.646521ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "85a9edee-feb2-4397-89eb-8b3d4180f52a", "content-length": "124", "date": "Wed, 15 May 2024 07:08:06 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "85a9edee-feb2-4397-89eb-8b3d4180f52a" } file = clients/ddm-admin-client/src/lib.rs:178
1842024-05-15T07:08:07.229ZINFOSledAgent (BackingFs): Stopping service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:170
1852024-05-15T07:08:07.324ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1862024-05-15T07:08:07.325ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 171.894585ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "36ca5c40-66b3-458a-b380-61b68a92cb9d", "content-length": "124", "date": "Wed, 15 May 2024 07:08:07 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "36ca5c40-66b3-458a-b380-61b68a92cb9d" } file = clients/ddm-admin-client/src/lib.rs:178
1872024-05-15T07:08:07.448ZINFOSledAgent (BackingFs): Mounting oxi_c3ed08c3-e0e1-43f2-b040-3c4755a2167c/backing/fmd on /var/fm/fmd file = sled-agent/src/backing_fs.rs:178
1882024-05-15T07:08:07.461ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/rsrc file = sled-agent/src/backing_fs.rs:186
1892024-05-15T07:08:07.461ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/ckpt file = sled-agent/src/backing_fs.rs:186
1902024-05-15T07:08:07.461ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/xprt file = sled-agent/src/backing_fs.rs:186
1912024-05-15T07:08:07.461ZINFOSledAgent (BackingFs): Starting service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:192
1922024-05-15T07:08:07.497ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1932024-05-15T07:08:07.498ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 95.619005ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "a004534c-1236-4bc3-bd6a-633835eb5c1f", "content-length": "124", "date": "Wed, 15 May 2024 07:08:07 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "a004534c-1236-4bc3-bd6a-633835eb5c1f" } file = clients/ddm-admin-client/src/lib.rs:178
1942024-05-15T07:08:07.595ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1952024-05-15T07:08:07.595ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 194.830069ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "bd9f34c8-02be-4b68-884d-0c320f377d64", "content-length": "124", "date": "Wed, 15 May 2024 07:08:07 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "bd9f34c8-02be-4b68-884d-0c320f377d64" } file = clients/ddm-admin-client/src/lib.rs:178
1962024-05-15T07:08:07.792ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1972024-05-15T07:08:07.793ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 333.055205ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "e67acb4d-63d5-4730-9a2a-fcffedf5e699", "content-length": "124", "date": "Wed, 15 May 2024 07:08:07 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "e67acb4d-63d5-4730-9a2a-fcffedf5e699" } file = clients/ddm-admin-client/src/lib.rs:178
1982024-05-15T07:08:08.127ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1992024-05-15T07:08:08.159ZINFOSledAgent: using '[AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }]' as data links for xde driver file = illumos-utils/src/opte/illumos.rs:90 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
2002024-05-15T07:08:08.160ZINFOSledAgent (dropshot): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:196 local_addr = [fd00:1122:3344:101::1]:55688 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
2012024-05-15T07:08:08.160ZINFOSledAgent (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
2022024-05-15T07:08:08.160ZINFOSledAgent (producer-server): starting oximeter metric producer server address = [fd00:1122:3344:101::1]:55688 file = oximeter/producer/src/lib.rs:284 interval = 30s producer_id = acb2a796-8e9f-4811-b601-726a12d79b0b route = //acb2a796-8e9f-4811-b601-726a12d79b0b sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
2032024-05-15T07:08:08.167ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 210.883973ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:393 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
2042024-05-15T07:08:08.208ZINFOSledAgent (VmmReservoirManager): 50% of 102952026112 physical ram = 51476013056 bytes) file = sled-agent/src/vmm_reservoir.rs:267 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
2052024-05-15T07:08:08.208ZINFOSledAgent (VmmReservoirManager): Setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:275 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
2062024-05-15T07:08:08.208ZINFOSledAgent (ServiceManager): sled agent started file = sled-agent/src/services.rs:993 underlay_address = fd00:1122:3344:101::1
2072024-05-15T07:08:08.208ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
2082024-05-15T07:08:08.209ZINFOSledAgent (dropshot (SledAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:196 local_addr = [fd00:1122:3344:101::1]:12345
2092024-05-15T07:08:08.209ZINFOSledAgent (SledAgentMain): Sled Agent loaded; recording configuration file = sled-agent/src/bootstrap/server.rs:430
2102024-05-15T07:08:08.209ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2112024-05-15T07:08:08.209ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2122024-05-15T07:08:08.210ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2132024-05-15T07:08:08.210ZINFOSledAgent (HardwareMonitor): Sled Agent Started file = sled-agent/src/hardware_monitor.rs:138
2142024-05-15T07:08:08.210ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:262 underlay_network_info = Some((fd00:1122:3344:101::2, Some(RackNetworkConfigV1 { rack_subnet: Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56 }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [PortConfigV1 { routes: [RouteConfig { destination: V4(Ipv4Network { addr: 0.0.0.0, prefix: 0 }), nexthop: 10.151.2.100, vlan_id: None }], addresses: [V4(Ipv4Network { addr: 10.151.2.174, prefix: 24 })], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false }], bgp: [], bfd: [] })))
2152024-05-15T07:08:08.210ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3617
2162024-05-15T07:08:08.210ZINFOSledAgent (SledAgentSprocketsServer): Connection closed file = sled-agent/src/bootstrap/sprockets_server.rs:73 remote_addr = [fdb0:18c0:4d0d:a02a::1]:53471
2172024-05-15T07:08:08.388ZINFOSledAgent (BootstrapAgentRssHandler): Peer agent initialized file = sled-agent/src/bootstrap/rss_handle.rs:108 peer = [fdb0:18c0:4d0d:a02a::1]:12346
2182024-05-15T07:08:08.388ZINFOSledAgent (BootstrapAgentRssHandler): Initialized sled agent file = sled-agent/src/bootstrap/rss_handle.rs:232 target_sled = [fdb0:18c0:4d0d:a02a::1]:12346
2192024-05-15T07:08:08.388ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2202024-05-15T07:08:08.389ZINFOSledAgent (RSS): No ledger in /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/rss-service-plan-v3.json file = common/src/ledger.rs:188
2212024-05-15T07:08:08.389ZINFOSledAgent (RSS): No ledger in /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/rss-service-plan-v3.json file = common/src/ledger.rs:188
2222024-05-15T07:08:08.389ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2232024-05-15T07:08:08.389ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2242024-05-15T07:08:08.390ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/acb2a796-8e9f-4811-b601-726a12d79b0b): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
2252024-05-15T07:08:08.390ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 464.359786ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:393 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
2262024-05-15T07:08:08.421ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42554
2272024-05-15T07:08:08.421ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2282024-05-15T07:08:08.421ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 164 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:42554 req_id = 54c00772-1052-4c33-8ab3-066a02009ede response_code = 200 uri = /inventory
2292024-05-15T07:08:08.453ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:37703
2302024-05-15T07:08:08.453ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 33 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:37703 req_id = 56c373a4-1220-4b81-a0a0-9544056e8f67 response_code = 200 uri = /sled-role
2312024-05-15T07:08:08.454ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2322024-05-15T07:08:08.454ZINFOSledAgent (RSS): Writing ledger to /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/.rss-service-plan-v3.json.tmp file = common/src/ledger.rs:199
2332024-05-15T07:08:08.455ZINFOSledAgent (RSS): Writing ledger to /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/.rss-service-plan-v3.json.tmp file = common/src/ledger.rs:199
2342024-05-15T07:08:08.455ZINFOSledAgent (RSS): Service plan written to storage file = sled-agent/src/rack_setup/plan/service.rs:815
2352024-05-15T07:08:08.486ZINFOSledAgent (RSS): attempting to set up sled's storage: OmicronPhysicalDisksConfig { disks: [OmicronPhysicalDiskConfig { id: bf2a0dce-f659-4a90-aa0b-dd0a03b87954, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, pool_id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool) }, OmicronPhysicalDiskConfig { id: 3da5cdb7-031b-441d-bcba-8297bc53c14f, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, pool_id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool) }, OmicronPhysicalDiskConfig { id: 2da42829-b77d-42b5-8212-48a387442b7c, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, pool_id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool) }, OmicronPhysicalDiskConfig { id: d27566ee-ab55-4b03-8f38-07e4c4675371, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, pool_id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool) }, OmicronPhysicalDiskConfig { id: 6c962dd2-23d0-4373-b1a1-9279cf19ae6d, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, pool_id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool) }, OmicronPhysicalDiskConfig { id: a3e4dcc9-7f51-462f-af2f-8c74c5b2c173, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, pool_id: 5da68ecb-08f6-4895-925f-903c81000441 (zpool) }, OmicronPhysicalDiskConfig { id: 4c36d026-3af1-492b-8432-acefaab135a6, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }, pool_id: 3b36f3f1-29d0-4d11-8b6b-ae2a89620d43 (zpool) }, OmicronPhysicalDiskConfig { id: 2150da11-52cd-42fe-9d41-aae1a3441958, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }, pool_id: 2780ea55-1799-4c3e-916a-9024b9586eaf (zpool) }, OmicronPhysicalDiskConfig { id: 575ec239-a350-4c7d-8ea3-b2a988f60446, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }, pool_id: bae019e4-4e95-466e-aa8a-f660660e26c1 (zpool) }], generation: Generation(1) } file = sled-agent/src/rack_setup/service.rs:347 sled_address = [fd00:1122:3344:101::1]:12345
2362024-05-15T07:08:08.486ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:52314
2372024-05-15T07:08:08.486ZINFOSledAgent (StorageManager): Received OmicronPhysicalDisksEnsure { config: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, id: bf2a0dce-f659-4a90-aa0b-dd0a03b87954, pool_id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, id: 3da5cdb7-031b-441d-bcba-8297bc53c14f, pool_id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, id: 2da42829-b77d-42b5-8212-48a387442b7c, pool_id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, id: d27566ee-ab55-4b03-8f38-07e4c4675371, pool_id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, id: 6c962dd2-23d0-4373-b1a1-9279cf19ae6d, pool_id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, id: a3e4dcc9-7f51-462f-af2f-8c74c5b2c173, pool_id: 5da68ecb-08f6-4895-925f-903c81000441 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }, id: 4c36d026-3af1-492b-8432-acefaab135a6, pool_id: 3b36f3f1-29d0-4d11-8b6b-ae2a89620d43 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }, id: 2150da11-52cd-42fe-9d41-aae1a3441958, pool_id: 2780ea55-1799-4c3e-916a-9024b9586eaf (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }, id: 575ec239-a350-4c7d-8ea3-b2a988f60446, pool_id: bae019e4-4e95-466e-aa8a-f660660e26c1 (zpool) }] }, tx: ... } file = sled-storage/src/manager.rs:381
2382024-05-15T07:08:08.486ZINFOSledAgent (StorageManager): No ledger in /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2392024-05-15T07:08:08.486ZINFOSledAgent (StorageManager): No ledger in /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2402024-05-15T07:08:08.486ZINFOSledAgent (StorageManager): No previously-stored 'requested disks', creating new ledger file = sled-storage/src/manager.rs:681 request = omicron_physical_disks_ensure
2412024-05-15T07:08:08.486ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:339
2422024-05-15T07:08:08.486ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
2432024-05-15T07:08:08.486ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
2442024-05-15T07:08:08.486ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:112 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } path = /scratch/u2_0.vdev
2452024-05-15T07:08:08.580ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_0.vdev file = sled-hardware/src/disk.rs:304
2462024-05-15T07:08:08.580ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool)
2472024-05-15T07:08:08.947ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 877.607412ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:393 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
2482024-05-15T07:08:09.212ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/acb2a796-8e9f-4811-b601-726a12d79b0b): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
2492024-05-15T07:08:09.590ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }
2502024-05-15T07:08:09.765ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
2512024-05-15T07:08:09.765ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
2522024-05-15T07:08:09.765ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
2532024-05-15T07:08:09.765ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
2542024-05-15T07:08:09.765ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2552024-05-15T07:08:09.765ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2562024-05-15T07:08:09.765ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
2572024-05-15T07:08:09.950ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 2.903407045s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:393 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
2582024-05-15T07:08:10.325ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_switch
2592024-05-15T07:08:10.519ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2602024-05-15T07:08:10.736ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/acb2a796-8e9f-4811-b601-726a12d79b0b): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
2612024-05-15T07:08:10.736ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/acb2a796-8e9f-4811-b601-726a12d79b0b): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
2622024-05-15T07:08:10.948ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_4e6ef30f-a7d9-4acc-99aa-59ae1084fa14/crypt/zone file = sled-storage/src/dataset.rs:395
2632024-05-15T07:08:11.135ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 61.742857ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2642024-05-15T07:08:11.669ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 112.265741ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2652024-05-15T07:08:12.281ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/acb2a796-8e9f-4811-b601-726a12d79b0b): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
2662024-05-15T07:08:12.281ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/acb2a796-8e9f-4811-b601-726a12d79b0b): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
2672024-05-15T07:08:12.281ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 114.940381ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2682024-05-15T07:08:12.785ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 482.068348ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2692024-05-15T07:08:12.964ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 3.84884602s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:393 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
2702024-05-15T07:08:13.614ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 file = sled-storage/src/dataset.rs:480
2712024-05-15T07:08:13.847ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/acb2a796-8e9f-4811-b601-726a12d79b0b): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
2722024-05-15T07:08:13.847ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.166310241s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2732024-05-15T07:08:13.853ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
2742024-05-15T07:08:13.853ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
2752024-05-15T07:08:13.854ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
2762024-05-15T07:08:13.854ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
2772024-05-15T07:08:13.854ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:112 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } path = /scratch/u2_1.vdev
2782024-05-15T07:08:14.091ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_1.vdev file = sled-hardware/src/disk.rs:304
2792024-05-15T07:08:14.091ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool)
2802024-05-15T07:08:15.280ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }
2812024-05-15T07:08:15.510ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
2822024-05-15T07:08:15.510ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
2832024-05-15T07:08:15.510ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
2842024-05-15T07:08:15.510ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
2852024-05-15T07:08:15.510ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2862024-05-15T07:08:15.510ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2872024-05-15T07:08:15.510ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
2882024-05-15T07:08:16.281ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/acb2a796-8e9f-4811-b601-726a12d79b0b): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
2892024-05-15T07:08:16.764ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/acb2a796-8e9f-4811-b601-726a12d79b0b): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
2902024-05-15T07:08:16.997ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 4.765630932s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:393 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
2912024-05-15T07:08:17.238ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2922024-05-15T07:08:17.495ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/acb2a796-8e9f-4811-b601-726a12d79b0b): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
2932024-05-15T07:08:17.754ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_12d428b9-eedd-40c0-8928-1d92a73f181d/crypt/zone file = sled-storage/src/dataset.rs:395
2942024-05-15T07:08:17.986ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/acb2a796-8e9f-4811-b601-726a12d79b0b): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
2952024-05-15T07:08:17.986ZINFOSledAgent (ServiceManager): Ensuring tfportrear0_0/ll exists in zone file = sled-agent/src/services.rs:2303
2962024-05-15T07:08:18.445ZINFOSledAgent (VmmReservoirManager): Finished setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:279 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
2972024-05-15T07:08:18.498ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/acb2a796-8e9f-4811-b601-726a12d79b0b): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
2982024-05-15T07:08:18.563ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_12d428b9-eedd-40c0-8928-1d92a73f181d file = sled-storage/src/dataset.rs:480
2992024-05-15T07:08:18.571ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
3002024-05-15T07:08:18.572ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
3012024-05-15T07:08:18.572ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
3022024-05-15T07:08:18.572ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3032024-05-15T07:08:18.572ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:112 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } path = /scratch/u2_2.vdev
3042024-05-15T07:08:18.582ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_2.vdev file = sled-hardware/src/disk.rs:304
3052024-05-15T07:08:18.582ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool)
3062024-05-15T07:08:18.678ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }
3072024-05-15T07:08:18.686ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
3082024-05-15T07:08:18.686ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
3092024-05-15T07:08:18.686ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
3102024-05-15T07:08:18.686ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
3112024-05-15T07:08:18.686ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3122024-05-15T07:08:18.686ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3132024-05-15T07:08:18.686ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3142024-05-15T07:08:18.721ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3152024-05-15T07:08:18.730ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_08d4507c-e72e-4afd-9f03-5e28f4ad0909/crypt/zone file = sled-storage/src/dataset.rs:395
3162024-05-15T07:08:18.855ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_08d4507c-e72e-4afd-9f03-5e28f4ad0909 file = sled-storage/src/dataset.rs:480
3172024-05-15T07:08:18.863ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
3182024-05-15T07:08:18.863ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
3192024-05-15T07:08:18.863ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
3202024-05-15T07:08:18.863ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3212024-05-15T07:08:18.863ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:112 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } path = /scratch/u2_3.vdev
3222024-05-15T07:08:18.873ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_3.vdev file = sled-hardware/src/disk.rs:304
3232024-05-15T07:08:18.873ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool)
3242024-05-15T07:08:18.966ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }
3252024-05-15T07:08:18.974ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
3262024-05-15T07:08:18.974ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
3272024-05-15T07:08:18.974ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
3282024-05-15T07:08:18.974ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
3292024-05-15T07:08:18.974ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3302024-05-15T07:08:18.974ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3312024-05-15T07:08:18.974ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3322024-05-15T07:08:19.011ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3332024-05-15T07:08:19.019ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_7ba4af02-bf8f-47c6-af79-d69fdad5d4bf/crypt/zone file = sled-storage/src/dataset.rs:395
3342024-05-15T07:08:19.145ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_7ba4af02-bf8f-47c6-af79-d69fdad5d4bf file = sled-storage/src/dataset.rs:480
3352024-05-15T07:08:19.153ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
3362024-05-15T07:08:19.153ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
3372024-05-15T07:08:19.153ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
3382024-05-15T07:08:19.153ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3392024-05-15T07:08:19.153ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:112 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } path = /scratch/u2_4.vdev
3402024-05-15T07:08:19.163ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_4.vdev file = sled-hardware/src/disk.rs:304
3412024-05-15T07:08:19.163ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool)
3422024-05-15T07:08:19.260ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }
3432024-05-15T07:08:19.268ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
3442024-05-15T07:08:19.268ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
3452024-05-15T07:08:19.268ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
3462024-05-15T07:08:19.268ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
3472024-05-15T07:08:19.268ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3482024-05-15T07:08:19.268ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3492024-05-15T07:08:19.268ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3502024-05-15T07:08:19.304ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3512024-05-15T07:08:19.311ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_62a187a2-0706-420e-a881-83ff781ad6c1/crypt/zone file = sled-storage/src/dataset.rs:395
3522024-05-15T07:08:19.436ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_62a187a2-0706-420e-a881-83ff781ad6c1 file = sled-storage/src/dataset.rs:480
3532024-05-15T07:08:19.445ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
3542024-05-15T07:08:19.445ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
3552024-05-15T07:08:19.445ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
3562024-05-15T07:08:19.445ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3572024-05-15T07:08:19.445ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:112 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } path = /scratch/u2_5.vdev
3582024-05-15T07:08:19.455ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_5.vdev file = sled-hardware/src/disk.rs:304
3592024-05-15T07:08:19.455ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 5da68ecb-08f6-4895-925f-903c81000441 (zpool)
3602024-05-15T07:08:19.555ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 5da68ecb-08f6-4895-925f-903c81000441 (zpool), kind: External }
3612024-05-15T07:08:19.562ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
3622024-05-15T07:08:19.562ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
3632024-05-15T07:08:19.562ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
3642024-05-15T07:08:19.562ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
3652024-05-15T07:08:19.562ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3662024-05-15T07:08:19.562ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3672024-05-15T07:08:19.562ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3682024-05-15T07:08:19.567ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/acb2a796-8e9f-4811-b601-726a12d79b0b): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
3692024-05-15T07:08:19.598ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3702024-05-15T07:08:19.606ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_5da68ecb-08f6-4895-925f-903c81000441/crypt/zone file = sled-storage/src/dataset.rs:395
3712024-05-15T07:08:19.730ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_5da68ecb-08f6-4895-925f-903c81000441 file = sled-storage/src/dataset.rs:480
3722024-05-15T07:08:19.738ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
3732024-05-15T07:08:19.738ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
3742024-05-15T07:08:19.738ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
3752024-05-15T07:08:19.738ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3762024-05-15T07:08:19.738ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:112 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } path = /scratch/u2_6.vdev
3772024-05-15T07:08:19.748ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_6.vdev file = sled-hardware/src/disk.rs:304
3782024-05-15T07:08:19.748ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 3b36f3f1-29d0-4d11-8b6b-ae2a89620d43 (zpool)
3792024-05-15T07:08:19.843ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 3b36f3f1-29d0-4d11-8b6b-ae2a89620d43 (zpool), kind: External }
3802024-05-15T07:08:19.851ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
3812024-05-15T07:08:19.851ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
3822024-05-15T07:08:19.851ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
3832024-05-15T07:08:19.851ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
3842024-05-15T07:08:19.851ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3852024-05-15T07:08:19.851ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3862024-05-15T07:08:19.851ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3872024-05-15T07:08:19.854ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/acb2a796-8e9f-4811-b601-726a12d79b0b): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
3882024-05-15T07:08:19.888ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3892024-05-15T07:08:19.897ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_3b36f3f1-29d0-4d11-8b6b-ae2a89620d43/crypt/zone file = sled-storage/src/dataset.rs:395
3902024-05-15T07:08:20.029ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_3b36f3f1-29d0-4d11-8b6b-ae2a89620d43 file = sled-storage/src/dataset.rs:480
3912024-05-15T07:08:20.038ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
3922024-05-15T07:08:20.038ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
3932024-05-15T07:08:20.038ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
3942024-05-15T07:08:20.038ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3952024-05-15T07:08:20.038ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:112 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } path = /scratch/u2_7.vdev
3962024-05-15T07:08:20.048ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_7.vdev file = sled-hardware/src/disk.rs:304
3972024-05-15T07:08:20.048ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 2780ea55-1799-4c3e-916a-9024b9586eaf (zpool)
3982024-05-15T07:08:20.141ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 2780ea55-1799-4c3e-916a-9024b9586eaf (zpool), kind: External }
3992024-05-15T07:08:20.147ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
4002024-05-15T07:08:20.148ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
4012024-05-15T07:08:20.148ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
4022024-05-15T07:08:20.148ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
4032024-05-15T07:08:20.148ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4042024-05-15T07:08:20.148ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4052024-05-15T07:08:20.148ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
4062024-05-15T07:08:20.185ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4072024-05-15T07:08:20.193ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_2780ea55-1799-4c3e-916a-9024b9586eaf/crypt/zone file = sled-storage/src/dataset.rs:395
4082024-05-15T07:08:20.314ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_2780ea55-1799-4c3e-916a-9024b9586eaf file = sled-storage/src/dataset.rs:480
4092024-05-15T07:08:20.322ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
4102024-05-15T07:08:20.322ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
4112024-05-15T07:08:20.322ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
4122024-05-15T07:08:20.322ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
4132024-05-15T07:08:20.322ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:112 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } path = /scratch/u2_8.vdev
4142024-05-15T07:08:20.331ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_8.vdev file = sled-hardware/src/disk.rs:304
4152024-05-15T07:08:20.331ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = bae019e4-4e95-466e-aa8a-f660660e26c1 (zpool)
4162024-05-15T07:08:20.426ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: bae019e4-4e95-466e-aa8a-f660660e26c1 (zpool), kind: External }
4172024-05-15T07:08:20.433ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
4182024-05-15T07:08:20.433ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
4192024-05-15T07:08:20.433ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
4202024-05-15T07:08:20.433ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
4212024-05-15T07:08:20.433ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4222024-05-15T07:08:20.433ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4232024-05-15T07:08:20.433ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
4242024-05-15T07:08:20.470ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4252024-05-15T07:08:20.478ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_bae019e4-4e95-466e-aa8a-f660660e26c1/crypt/zone file = sled-storage/src/dataset.rs:395
4262024-05-15T07:08:20.569ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/acb2a796-8e9f-4811-b601-726a12d79b0b): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
4272024-05-15T07:08:20.602ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_bae019e4-4e95-466e-aa8a-f660660e26c1 file = sled-storage/src/dataset.rs:480
4282024-05-15T07:08:20.610ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
4292024-05-15T07:08:20.610ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
4302024-05-15T07:08:20.610ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4312024-05-15T07:08:20.610ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: c3ed08c3-e0e1-43f2-b040-3c4755a2167c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 }, zpool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 3907cd6d-3c6d-4543-92de-9ec01f9bc32c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 }, zpool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 }, zpool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 }, zpool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, variant: U2, slot: 1030 }, zpool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, variant: U2, slot: 1031 }, zpool_name: ZpoolName { id: 5da68ecb-08f6-4895-925f-903c81000441 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }, variant: U2, slot: 1032 }, zpool_name: ZpoolName { id: 3b36f3f1-29d0-4d11-8b6b-ae2a89620d43 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }, variant: U2, slot: 1033 }, zpool_name: ZpoolName { id: 2780ea55-1799-4c3e-916a-9024b9586eaf (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }, variant: U2, slot: 1034 }, zpool_name: ZpoolName { id: bae019e4-4e95-466e-aa8a-f660660e26c1 (zpool), kind: External } }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
4322024-05-15T07:08:20.610ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
4332024-05-15T07:08:20.610ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
4342024-05-15T07:08:20.610ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4352024-05-15T07:08:20.611ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 12124443 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52314 req_id = a8bd8eae-8392-415b-b726-51516711c801 response_code = 200 uri = /omicron-physical-disks
4362024-05-15T07:08:20.641ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(2), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
4372024-05-15T07:08:20.641ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:50895
4382024-05-15T07:08:20.642ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
4392024-05-15T07:08:20.642ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4402024-05-15T07:08:20.693ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
4412024-05-15T07:08:20.693ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
4422024-05-15T07:08:20.745ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
4432024-05-15T07:08:20.745ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4442024-05-15T07:08:20.797ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4452024-05-15T07:08:20.797ZINFOSledAgent (ServiceManager): No ledger in /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:188
4462024-05-15T07:08:20.797ZINFOSledAgent (ServiceManager): No ledger in /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:188
4472024-05-15T07:08:20.797ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4482024-05-15T07:08:20.842ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4492024-05-15T07:08:20.842ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4502024-05-15T07:08:20.842ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4512024-05-15T07:08:20.896ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_adb60b2d-67ac-445a-817b-3e6611ca7982 file = illumos-utils/src/zone.rs:315
4522024-05-15T07:08:20.900ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_08569023-bf34-46c3-a27b-d9b75bdbd67c file = illumos-utils/src/zone.rs:315
4532024-05-15T07:08:20.904ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62 file = illumos-utils/src/zone.rs:315
4542024-05-15T07:08:20.917ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_adb60b2d-67ac-445a-817b-3e6611ca7982 file = illumos-utils/src/zone.rs:354
4552024-05-15T07:08:20.921ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_08569023-bf34-46c3-a27b-d9b75bdbd67c file = illumos-utils/src/zone.rs:354
4562024-05-15T07:08:20.925ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62 file = illumos-utils/src/zone.rs:354
4572024-05-15T07:08:21.571ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/acb2a796-8e9f-4811-b601-726a12d79b0b): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
4582024-05-15T07:08:21.764ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 15.454121794s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:393 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
4592024-05-15T07:08:21.853ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/acb2a796-8e9f-4811-b601-726a12d79b0b): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
4602024-05-15T07:08:22.442ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_adb60b2d-67ac-445a-817b-3e6611ca7982: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService1"/> <propval type="astring" name="gateway" value="fd00:1122:3344:2::2"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:2::1"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:2::1]:5353"/> <propval type="astring" name="dns_address" value="[fd00:1122:3344:2::1]:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4612024-05-15T07:08:22.442ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:87 prefix = Ipv6Prefix { addr: fd00:1122:3344:2::, len: 64 }
4622024-05-15T07:08:22.442ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_internal_dns_adb60b2d-67ac-445a-817b-3e6611ca7982
4632024-05-15T07:08:22.929ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_08569023-bf34-46c3-a27b-d9b75bdbd67c: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService2"/> <propval type="astring" name="gateway" value="fd00:1122:3344:3::2"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:3::1"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:3::1]:5353"/> <propval type="astring" name="dns_address" value="[fd00:1122:3344:3::1]:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4642024-05-15T07:08:23.416ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:87 prefix = Ipv6Prefix { addr: fd00:1122:3344:3::, len: 64 }
4652024-05-15T07:08:23.417ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService3"/> <propval type="astring" name="gateway" value="fd00:1122:3344:1::2"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:1::1"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:1::1]:5353"/> <propval type="astring" name="dns_address" value="[fd00:1122:3344:1::1]:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4662024-05-15T07:08:23.417ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:87 prefix = Ipv6Prefix { addr: fd00:1122:3344:1::, len: 64 }
4672024-05-15T07:08:23.425ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_adb60b2d-67ac-445a-817b-3e6611ca7982") failed: Property not found. retry in 69.341009ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_adb60b2d-67ac-445a-817b-3e6611ca7982
4682024-05-15T07:08:23.425ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_internal_dns_08569023-bf34-46c3-a27b-d9b75bdbd67c
4692024-05-15T07:08:23.429ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_internal_dns_18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62
4702024-05-15T07:08:23.504ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_adb60b2d-67ac-445a-817b-3e6611ca7982") failed: Property not found. retry in 138.27505ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_adb60b2d-67ac-445a-817b-3e6611ca7982
4712024-05-15T07:08:23.549ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_08569023-bf34-46c3-a27b-d9b75bdbd67c") failed: Property not found. retry in 66.1532ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_08569023-bf34-46c3-a27b-d9b75bdbd67c
4722024-05-15T07:08:23.555ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62") failed: Property not found. retry in 34.756274ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62
4732024-05-15T07:08:23.597ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62") failed: Property not found. retry in 134.171434ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62
4742024-05-15T07:08:23.702ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_08569023-bf34-46c3-a27b-d9b75bdbd67c") failed: Property not found. retry in 120.441799ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_08569023-bf34-46c3-a27b-d9b75bdbd67c
4752024-05-15T07:08:23.759ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_adb60b2d-67ac-445a-817b-3e6611ca7982") failed: Property not found. retry in 273.14767ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_adb60b2d-67ac-445a-817b-3e6611ca7982
4762024-05-15T07:08:23.877ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62") failed: Property not found. retry in 208.055766ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62
4772024-05-15T07:08:23.898ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_08569023-bf34-46c3-a27b-d9b75bdbd67c") failed: Property not found. retry in 109.485985ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_08569023-bf34-46c3-a27b-d9b75bdbd67c
4782024-05-15T07:08:24.027ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_08569023-bf34-46c3-a27b-d9b75bdbd67c") failed: Property not found. retry in 571.647691ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_08569023-bf34-46c3-a27b-d9b75bdbd67c
4792024-05-15T07:08:24.042ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_adb60b2d-67ac-445a-817b-3e6611ca7982") failed: Property not found. retry in 300.448797ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_adb60b2d-67ac-445a-817b-3e6611ca7982
4802024-05-15T07:08:24.096ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62") failed: Property not found. retry in 480.014683ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62
4812024-05-15T07:08:24.350ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_adb60b2d-67ac-445a-817b-3e6611ca7982") failed: Property not found. retry in 506.309752ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_adb60b2d-67ac-445a-817b-3e6611ca7982
4822024-05-15T07:08:24.586ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62") failed: Property not found. retry in 952.571801ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62
4832024-05-15T07:08:24.608ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_08569023-bf34-46c3-a27b-d9b75bdbd67c") failed: Property not found. retry in 1.160549834s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_08569023-bf34-46c3-a27b-d9b75bdbd67c
4842024-05-15T07:08:24.869ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_adb60b2d-67ac-445a-817b-3e6611ca7982") failed: Property not found. retry in 1.485934287s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_adb60b2d-67ac-445a-817b-3e6611ca7982
4852024-05-15T07:08:25.550ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62") failed: Property not found. retry in 886.927132ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62
4862024-05-15T07:08:25.778ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_08569023-bf34-46c3-a27b-d9b75bdbd67c") failed: Property not found. retry in 1.17524673s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_08569023-bf34-46c3-a27b-d9b75bdbd67c
4872024-05-15T07:08:26.021ZINFOSledAgent (ServiceManager): Ensuring bootstrap address fdb0:18c0:4d0d:a02a::2 exists in switch zone file = sled-agent/src/services.rs:2320
4882024-05-15T07:08:26.021ZINFOSledAgent (ServiceManager): Adding bootstrap address file = illumos-utils/src/running_zone.rs:626 zone = oxz_switch
4892024-05-15T07:08:26.364ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_adb60b2d-67ac-445a-817b-3e6611ca7982") failed: Property not found. retry in 754.976855ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_adb60b2d-67ac-445a-817b-3e6611ca7982
4902024-05-15T07:08:26.445ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62") failed: Property not found. retry in 580.641515ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62
4912024-05-15T07:08:26.568ZINFOSledAgent (ServiceManager): Forwarding bootstrap traffic via oxBootstrap0 to fe80::8:20ff:fedf:4abf file = sled-agent/src/services.rs:2327
4922024-05-15T07:08:26.573ZINFOSledAgent (ServiceManager): Zone not using gateway (even though underlay is up) file = sled-agent/src/services.rs:2388
4932024-05-15T07:08:26.691ZINFOSledAgent (ServiceManager): Setting up dendrite service file = sled-agent/src/services.rs:2573
4942024-05-15T07:08:26.931ZINFOSledAgent (ServiceManager): Setting up lldpd service file = sled-agent/src/services.rs:2737
4952024-05-15T07:08:26.964ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_08569023-bf34-46c3-a27b-d9b75bdbd67c") failed: Property not found. retry in 1.058980197s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_08569023-bf34-46c3-a27b-d9b75bdbd67c
4962024-05-15T07:08:27.034ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62") failed: Property not found. retry in 1.194010768s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62
4972024-05-15T07:08:27.124ZINFOSledAgent (ServiceManager): Setting up MGS service file = sled-agent/src/services.rs:2452
4982024-05-15T07:08:27.128ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_adb60b2d-67ac-445a-817b-3e6611ca7982") failed: Property not found. retry in 696.935222ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_adb60b2d-67ac-445a-817b-3e6611ca7982
4992024-05-15T07:08:27.427ZINFOSledAgent (ServiceManager): Setting up wicketd service file = sled-agent/src/services.rs:2487
5002024-05-15T07:08:27.647ZINFOSledAgent (ServiceManager): Setting up mgd service file = sled-agent/src/services.rs:2787
5012024-05-15T07:08:27.831ZINFOSledAgent (ServiceManager): Setting up mg-ddm service file = sled-agent/src/services.rs:2814
5022024-05-15T07:08:27.835ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_adb60b2d-67ac-445a-817b-3e6611ca7982") failed: Property not found. retry in 1.129063318s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_adb60b2d-67ac-445a-817b-3e6611ca7982
5032024-05-15T07:08:28.034ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_08569023-bf34-46c3-a27b-d9b75bdbd67c") failed: Property not found. retry in 841.067879ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_08569023-bf34-46c3-a27b-d9b75bdbd67c
5042024-05-15T07:08:28.048ZINFOSledAgent (ServiceManager): Setting up tfport service file = sled-agent/src/services.rs:2681
5052024-05-15T07:08:28.235ZINFOSledAgent (ServiceManager): Setting up Simulated SP service file = sled-agent/src/services.rs:2481
5062024-05-15T07:08:28.238ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62") failed: Property not found. retry in 1.113305531s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62
5072024-05-15T07:08:28.245ZINFOSledAgent (ServiceManager): Re-enabling running switch zone (new address) file = sled-agent/src/services.rs:3873 new = [fd00:1122:3344:101::2, ::1] old = [::1]
5082024-05-15T07:08:28.245ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists file = sled-agent/src/services.rs:3889
5092024-05-15T07:08:28.246ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::2, prefix: 64 })) file = illumos-utils/src/running_zone.rs:609 zone = oxz_switch
5102024-05-15T07:08:28.801ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists - OK file = sled-agent/src/services.rs:3897
5112024-05-15T07:08:28.808ZINFOSledAgent (ServiceManager): configuring dendrite service file = sled-agent/src/services.rs:3953
5122024-05-15T07:08:28.967ZINFOSledAgent (ServiceManager): configuring lldp service file = sled-agent/src/services.rs:4015
5132024-05-15T07:08:29.067ZINFOSledAgent (ServiceManager): configuring wicketd file = sled-agent/src/services.rs:3996 rack_subnet = fd00:1122:3344::
5142024-05-15T07:08:29.088ZINFOSledAgent (ServiceManager): configuring mgd service file = sled-agent/src/services.rs:4043
5152024-05-15T07:08:29.184ZINFOSledAgent (ServiceManager): configuring mg-ddm service file = sled-agent/src/services.rs:4070
5162024-05-15T07:08:29.308ZINFOSledAgent (ServiceManager): Determining physical location of our switch zone at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:365
5172024-05-15T07:08:29.340ZINFOSledAgent (ServiceManager): Initializing 1 Uplinks on Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:413
5182024-05-15T07:08:29.370ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_internal_dns_adb60b2d-67ac-445a-817b-3e6611ca7982
5192024-05-15T07:08:29.370ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_internal_dns_08569023-bf34-46c3-a27b-d9b75bdbd67c
5202024-05-15T07:08:29.370ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_internal_dns_18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62
5212024-05-15T07:08:29.370ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5222024-05-15T07:08:29.371ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5232024-05-15T07:08:29.371ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 8729531 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:50895 req_id = 8908c98c-00ab-42cf-93e4-be0163903f21 response_code = 204 uri = /omicron-zones
5242024-05-15T07:08:29.373ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:1::1]:5353 file = sled-agent/src/rack_setup/service.rs:548
5252024-05-15T07:08:29.379ZINFOSledAgent (ServiceManager): Building Port Configuration file = sled-agent/src/bootstrap/early_networking.rs:651
5262024-05-15T07:08:29.379ZINFOSledAgent (ServiceManager): Checking dendrite uptime file = sled-agent/src/bootstrap/early_networking.rs:689
5272024-05-15T07:08:29.379ZINFOSledAgent (ServiceManager): Dendrite online file = sled-agent/src/bootstrap/early_networking.rs:692 uptime = 3
5282024-05-15T07:08:29.379ZINFOSledAgent (ServiceManager): Configuring default uplink on switch config = PortSettings {\n links: {\n "0": LinkSettings {\n addrs: [\n 10.151.2.174,\n ],\n params: LinkCreate {\n autoneg: false,\n fec: None,\n kr: false,\n lane: Some(\n LinkId(\n 0,\n ),\n ),\n speed: Speed40G,\n },\n },\n },\n} file = sled-agent/src/bootstrap/early_networking.rs:435
5292024-05-15T07:08:29.426ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:2::1]:5353 file = sled-agent/src/rack_setup/service.rs:548
5302024-05-15T07:08:29.462ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:381
5312024-05-15T07:08:29.462ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }
5322024-05-15T07:08:29.462ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }
5332024-05-15T07:08:29.462ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }
5342024-05-15T07:08:29.462ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }
5352024-05-15T07:08:29.462ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }
5362024-05-15T07:08:29.462ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }
5372024-05-15T07:08:29.462ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }
5382024-05-15T07:08:29.462ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }
5392024-05-15T07:08:29.462ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }
5402024-05-15T07:08:29.462ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }
5412024-05-15T07:08:29.462ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }
5422024-05-15T07:08:29.462ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }
5432024-05-15T07:08:29.462ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }
5442024-05-15T07:08:29.463ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }
5452024-05-15T07:08:29.463ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }
5462024-05-15T07:08:29.463ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }
5472024-05-15T07:08:29.463ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }
5482024-05-15T07:08:29.463ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }
5492024-05-15T07:08:29.463ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }
5502024-05-15T07:08:29.463ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }
5512024-05-15T07:08:29.463ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }
5522024-05-15T07:08:29.463ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }
5532024-05-15T07:08:29.490ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:3::1]:5353 file = sled-agent/src/rack_setup/service.rs:548
5542024-05-15T07:08:29.537ZINFOSledAgent (RSS): Configured all DNS servers file = sled-agent/src/rack_setup/service.rs:584
5552024-05-15T07:08:29.537ZINFOSledAgent (RSS): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:259
5562024-05-15T07:08:29.569ZINFOSledAgent (RSS): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:276
5572024-05-15T07:08:29.569ZINFOSledAgent (RSS): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:301
5582024-05-15T07:08:29.569ZINFOSledAgent (RSS): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:193
5592024-05-15T07:08:29.598ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(3), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 952adea2-114c-4099-9bd9-86f261fe9687, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f2afc369-fa05-4e43-9c24-aaf26564f459, kind: Service { id: 952adea2-114c-4099-9bd9-86f261fe9687 }, name: Name("ntp-952adea2-114c-4099-9bd9-86f261fe9687"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 18])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
5602024-05-15T07:08:29.598ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:49467
5612024-05-15T07:08:29.598ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
5622024-05-15T07:08:29.598ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5632024-05-15T07:08:29.619ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
5642024-05-15T07:08:29.619ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
5652024-05-15T07:08:29.636ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
5662024-05-15T07:08:29.636ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5672024-05-15T07:08:29.655ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
5682024-05-15T07:08:29.655ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
5692024-05-15T07:08:29.655ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
5702024-05-15T07:08:29.655ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
5712024-05-15T07:08:29.665ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:259
5722024-05-15T07:08:29.697ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:276
5732024-05-15T07:08:29.697ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:301
5742024-05-15T07:08:29.698ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:193
5752024-05-15T07:08:29.742ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:394 port = Port { inner: PortInner { name: "opte0", ip: 172.30.3.5, mac: MacAddr6([168, 64, 37, 255, 138, 18]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.3.1 }, vnic: "vopte0" } }
5762024-05-15T07:08:29.742ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1290 zone_type = ntp
5772024-05-15T07:08:29.742ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.2.169, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
5782024-05-15T07:08:29.742ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.2.169, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
5792024-05-15T07:08:29.743ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.2.169, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
5802024-05-15T07:08:29.744ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
5812024-05-15T07:08:29.766ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_ntp_952adea2-114c-4099-9bd9-86f261fe9687 file = illumos-utils/src/zone.rs:315
5822024-05-15T07:08:29.786ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_ntp_952adea2-114c-4099-9bd9-86f261fe9687 file = illumos-utils/src/zone.rs:354
5832024-05-15T07:08:30.416ZINFOSledAgent (ServiceManager): Profile for oxz_ntp_952adea2-114c-4099-9bd9-86f261fe9687: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService4"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::1b"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/chrony-setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="allow" value="fd00:1122:3344:100::/56"/> <propval type="boolean" name="boundary" value="true"/> <propval type="astring" name="server" value="0.pool.ntp.org"/> </property_group> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="1.1.1.1" /> <value_node value="9.9.9.9" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/ntp"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte0"/> <propval type="astring" name="gateway" value="172.30.3.1"/> <propval type="astring" name="ip" value="172.30.3.5"/> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
5842024-05-15T07:08:30.416ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_ntp_952adea2-114c-4099-9bd9-86f261fe9687
5852024-05-15T07:08:30.557ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_952adea2-114c-4099-9bd9-86f261fe9687") failed: Property not found. retry in 30.606696ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_952adea2-114c-4099-9bd9-86f261fe9687
5862024-05-15T07:08:30.595ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_952adea2-114c-4099-9bd9-86f261fe9687") failed: Property not found. retry in 87.265972ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_952adea2-114c-4099-9bd9-86f261fe9687
5872024-05-15T07:08:30.695ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_952adea2-114c-4099-9bd9-86f261fe9687") failed: Property not found. retry in 126.830481ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_952adea2-114c-4099-9bd9-86f261fe9687
5882024-05-15T07:08:30.833ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_952adea2-114c-4099-9bd9-86f261fe9687") failed: Property not found. retry in 332.146718ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_952adea2-114c-4099-9bd9-86f261fe9687
5892024-05-15T07:08:31.175ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_952adea2-114c-4099-9bd9-86f261fe9687") failed: Property not found. retry in 1.031646728s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_952adea2-114c-4099-9bd9-86f261fe9687
5902024-05-15T07:08:32.218ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_952adea2-114c-4099-9bd9-86f261fe9687") failed: Property not found. retry in 894.92526ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_952adea2-114c-4099-9bd9-86f261fe9687
5912024-05-15T07:08:32.731ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:61118
5922024-05-15T07:08:32.732ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 46 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:61118 req_id = b6a26b1f-9489-4234-9b3f-ab52232fde61 response_code = 200 uri = /baseboard
5932024-05-15T07:08:33.123ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_952adea2-114c-4099-9bd9-86f261fe9687") failed: Property not found. retry in 1.341077672s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_952adea2-114c-4099-9bd9-86f261fe9687
5942024-05-15T07:08:34.474ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_952adea2-114c-4099-9bd9-86f261fe9687") failed: Property not found. retry in 770.329619ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_952adea2-114c-4099-9bd9-86f261fe9687
5952024-05-15T07:08:35.254ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_952adea2-114c-4099-9bd9-86f261fe9687") failed: Property not found. retry in 1.440427843s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_952adea2-114c-4099-9bd9-86f261fe9687
5962024-05-15T07:08:36.715ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_ntp_952adea2-114c-4099-9bd9-86f261fe9687
5972024-05-15T07:08:36.715ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5982024-05-15T07:08:36.715ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5992024-05-15T07:08:36.715ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 7117508 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49467 req_id = afebd678-1fa6-4c83-8ea9-e243045880c4 response_code = 204 uri = /omicron-zones
6002024-05-15T07:08:36.717ZINFOSledAgent (RSS): Waiting for rack time synchronization file = sled-agent/src/rack_setup/service.rs:625
6012024-05-15T07:08:36.746ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:605
6022024-05-15T07:08:36.746ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:59323
6032024-05-15T07:08:36.755ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_952adea2-114c-4099-9bd9-86f261fe9687': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3603
6042024-05-15T07:08:36.755ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 8582 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:59323 req_id = 7f790f91-d603-4eab-99d6-12b3b699c435 response_code = 500 uri = /timesync
6052024-05-15T07:08:36.758ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:655
6062024-05-15T07:08:36.851ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:605
6072024-05-15T07:08:36.851ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:35565
6082024-05-15T07:08:40.859ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_952adea2-114c-4099-9bd9-86f261fe9687': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3603
6092024-05-15T07:08:40.859ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 4007967 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:35565 req_id = cbc2d379-ccba-419d-bd67-7fe835760aca response_code = 500 uri = /timesync
6102024-05-15T07:08:40.864ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:655
6112024-05-15T07:08:41.012ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:605
6122024-05-15T07:08:41.012ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:50742
6132024-05-15T07:08:41.021ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 9433 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:50742 req_id = 77075011-956a-497b-b697-aedd8324854b response_code = 200 uri = /timesync
6142024-05-15T07:08:41.024ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:633
6152024-05-15T07:08:41.024ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:655
6162024-05-15T07:08:41.257ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:605
6172024-05-15T07:08:41.257ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:48949
6182024-05-15T07:08:41.266ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 8873 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:48949 req_id = 2e190019-5674-4a9a-97dd-315d68a90b50 response_code = 200 uri = /timesync
6192024-05-15T07:08:41.269ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:633
6202024-05-15T07:08:41.269ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:655
6212024-05-15T07:08:41.425ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:605
6222024-05-15T07:08:41.425ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:51419
6232024-05-15T07:08:41.433ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 7534 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:51419 req_id = 08566412-3cb2-4b07-a06f-12c1b915bfda response_code = 200 uri = /timesync
6242024-05-15T07:08:41.436ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:633
6252024-05-15T07:08:41.436ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:655
6262024-05-15T07:08:41.595ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:605
6272024-05-15T07:08:41.595ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:41619
6282024-05-15T07:08:41.602ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 7686 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:41619 req_id = a3d10a62-b7c8-4bdc-beeb-8ebc98dc75dc response_code = 200 uri = /timesync
6292024-05-15T07:08:41.605ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:633
6302024-05-15T07:08:41.605ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:655
6312024-05-15T07:08:41.956ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:605
6322024-05-15T07:08:41.956ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:61670
6332024-05-15T07:08:41.965ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 8441 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:61670 req_id = 02b96f3f-879f-41d4-a54f-364aa12a70dc response_code = 200 uri = /timesync
6342024-05-15T07:08:41.966ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:633
6352024-05-15T07:08:41.967ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:655
6362024-05-15T07:08:42.332ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:605
6372024-05-15T07:08:42.332ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:53088
6382024-05-15T07:08:42.340ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 8429 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:53088 req_id = 99df8206-c000-43e6-8edc-1e2e34ef9f1f response_code = 200 uri = /timesync
6392024-05-15T07:08:42.342ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:633
6402024-05-15T07:08:42.342ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:655
6412024-05-15T07:08:42.582ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:605
6422024-05-15T07:08:42.583ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:46420
6432024-05-15T07:08:42.590ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 7651 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:46420 req_id = c0986356-7ae9-4ac3-984d-67b2857842ee response_code = 200 uri = /timesync
6442024-05-15T07:08:42.592ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:633
6452024-05-15T07:08:42.592ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:655
6462024-05-15T07:08:43.150ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:605
6472024-05-15T07:08:43.150ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:45241
6482024-05-15T07:08:43.158ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 8094 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:45241 req_id = 6b5aa85e-4891-4447-87bc-c9eab6ffb868 response_code = 200 uri = /timesync
6492024-05-15T07:08:43.160ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:633
6502024-05-15T07:08:43.160ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:655
6512024-05-15T07:08:43.889ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:605
6522024-05-15T07:08:43.889ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:56708
6532024-05-15T07:08:43.898ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 9021 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:56708 req_id = 19348d94-f53f-4b74-af5e-beeb5c819957 response_code = 200 uri = /timesync
6542024-05-15T07:08:43.900ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:633
6552024-05-15T07:08:43.900ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:655
6562024-05-15T07:08:44.767ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:605
6572024-05-15T07:08:44.768ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:64487
6582024-05-15T07:08:44.776ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 7976 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:64487 req_id = 9b09a1cf-5ccc-4013-8c98-bf16360a01b0 response_code = 200 uri = /timesync
6592024-05-15T07:08:44.777ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:633
6602024-05-15T07:08:44.777ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:655
6612024-05-15T07:08:45.262ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:605
6622024-05-15T07:08:45.262ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:35742
6632024-05-15T07:08:45.283ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 20393 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:35742 req_id = c5e3b249-bd4f-41c0-ada2-6762ca43e27e response_code = 200 uri = /timesync
6642024-05-15T07:08:45.286ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: true, ref_id: 1163513699, ip_addr: 69.89.207.99, stratum: 2, ref_time: 1715756925.238036, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:633
6652024-05-15T07:08:45.286ZINFOSledAgent (RSS): Finished setting up Internal DNS and NTP file = sled-agent/src/rack_setup/service.rs:1216
6662024-05-15T07:08:45.316ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 952adea2-114c-4099-9bd9-86f261fe9687, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f2afc369-fa05-4e43-9c24-aaf26564f459, kind: Service { id: 952adea2-114c-4099-9bd9-86f261fe9687 }, name: Name("ntp-952adea2-114c-4099-9bd9-86f261fe9687"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 18])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } } } }, OmicronZoneConfig { id: c7d123b0-1fe2-4336-ad94-79216ae10c14, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9035da45-fa42-447f-985d-89ad334749a3, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } } } }, OmicronZoneConfig { id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
6672024-05-15T07:08:45.316ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:46219
6682024-05-15T07:08:45.316ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
6692024-05-15T07:08:45.316ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6702024-05-15T07:08:45.337ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
6712024-05-15T07:08:45.337ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
6722024-05-15T07:08:45.356ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
6732024-05-15T07:08:45.356ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6742024-05-15T07:08:45.376ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6752024-05-15T07:08:45.376ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6762024-05-15T07:08:45.436ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6772024-05-15T07:08:45.436ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6782024-05-15T07:08:45.488ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6792024-05-15T07:08:45.488ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6802024-05-15T07:08:45.540ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6812024-05-15T07:08:45.540ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6822024-05-15T07:08:45.591ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6832024-05-15T07:08:45.591ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6842024-05-15T07:08:45.640ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
6852024-05-15T07:08:45.640ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
6862024-05-15T07:08:45.640ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
6872024-05-15T07:08:45.641ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
6882024-05-15T07:08:45.673ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
6892024-05-15T07:08:45.673ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
6902024-05-15T07:08:45.673ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
6912024-05-15T07:08:45.673ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
6922024-05-15T07:08:45.673ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
6932024-05-15T07:08:45.673ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3", TimeNotSynchronized), ("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9", TimeNotSynchronized), ("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015", TimeNotSynchronized), ("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14", TimeNotSynchronized), ("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 356992 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46219 req_id = cb805569-83a4-4e8b-800a-1a73ad54a20b response_code = 500 uri = /omicron-zones
6942024-05-15T07:08:45.673ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "cb805569-83a4-4e8b-800a-1a73ad54a20b", "content-length": "124", "date": "Wed, 15 May 2024 07:08:45 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "cb805569-83a4-4e8b-800a-1a73ad54a20b" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 65.832012ms sled_address = [fd00:1122:3344:101::1]:12345
6952024-05-15T07:08:45.741ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 952adea2-114c-4099-9bd9-86f261fe9687, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f2afc369-fa05-4e43-9c24-aaf26564f459, kind: Service { id: 952adea2-114c-4099-9bd9-86f261fe9687 }, name: Name("ntp-952adea2-114c-4099-9bd9-86f261fe9687"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 18])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } } } }, OmicronZoneConfig { id: c7d123b0-1fe2-4336-ad94-79216ae10c14, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9035da45-fa42-447f-985d-89ad334749a3, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } } } }, OmicronZoneConfig { id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
6962024-05-15T07:08:45.742ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
6972024-05-15T07:08:45.742ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6982024-05-15T07:08:45.758ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
6992024-05-15T07:08:45.758ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
7002024-05-15T07:08:45.777ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
7012024-05-15T07:08:45.777ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7022024-05-15T07:08:45.795ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7032024-05-15T07:08:45.795ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7042024-05-15T07:08:45.814ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7052024-05-15T07:08:45.814ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7062024-05-15T07:08:45.833ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7072024-05-15T07:08:45.833ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7082024-05-15T07:08:45.851ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7092024-05-15T07:08:45.851ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7102024-05-15T07:08:45.870ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7112024-05-15T07:08:45.870ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7122024-05-15T07:08:45.889ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
7132024-05-15T07:08:45.889ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
7142024-05-15T07:08:45.889ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
7152024-05-15T07:08:45.889ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
7162024-05-15T07:08:45.922ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
7172024-05-15T07:08:45.922ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
7182024-05-15T07:08:45.922ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
7192024-05-15T07:08:45.922ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
7202024-05-15T07:08:45.922ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
7212024-05-15T07:08:45.922ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3", TimeNotSynchronized), ("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015", TimeNotSynchronized), ("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9", TimeNotSynchronized), ("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14", TimeNotSynchronized), ("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 180698 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46219 req_id = 960bdd3e-303e-46cc-8ea6-7fb7fb14fa4f response_code = 500 uri = /omicron-zones
7222024-05-15T07:08:45.922ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "960bdd3e-303e-46cc-8ea6-7fb7fb14fa4f", "content-length": "124", "date": "Wed, 15 May 2024 07:08:45 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "960bdd3e-303e-46cc-8ea6-7fb7fb14fa4f" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 149.706127ms sled_address = [fd00:1122:3344:101::1]:12345
7232024-05-15T07:08:46.073ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 952adea2-114c-4099-9bd9-86f261fe9687, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f2afc369-fa05-4e43-9c24-aaf26564f459, kind: Service { id: 952adea2-114c-4099-9bd9-86f261fe9687 }, name: Name("ntp-952adea2-114c-4099-9bd9-86f261fe9687"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 18])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } } } }, OmicronZoneConfig { id: c7d123b0-1fe2-4336-ad94-79216ae10c14, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9035da45-fa42-447f-985d-89ad334749a3, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } } } }, OmicronZoneConfig { id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
7242024-05-15T07:08:46.073ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
7252024-05-15T07:08:46.073ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7262024-05-15T07:08:46.094ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
7272024-05-15T07:08:46.094ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
7282024-05-15T07:08:46.114ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
7292024-05-15T07:08:46.114ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7302024-05-15T07:08:46.134ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7312024-05-15T07:08:46.134ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7322024-05-15T07:08:46.152ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7332024-05-15T07:08:46.152ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7342024-05-15T07:08:46.170ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7352024-05-15T07:08:46.170ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7362024-05-15T07:08:46.189ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7372024-05-15T07:08:46.190ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7382024-05-15T07:08:46.208ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7392024-05-15T07:08:46.208ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7402024-05-15T07:08:46.226ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
7412024-05-15T07:08:46.226ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
7422024-05-15T07:08:46.226ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
7432024-05-15T07:08:46.226ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
7442024-05-15T07:08:46.260ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
7452024-05-15T07:08:46.260ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
7462024-05-15T07:08:46.260ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
7472024-05-15T07:08:46.260ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
7482024-05-15T07:08:46.260ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
7492024-05-15T07:08:46.260ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9", TimeNotSynchronized), ("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364", TimeNotSynchronized), ("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015", TimeNotSynchronized), ("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3", TimeNotSynchronized), ("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 186852 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46219 req_id = 35d250df-3018-433a-b9b2-f1a02f4ed89f response_code = 500 uri = /omicron-zones
7502024-05-15T07:08:46.260ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "35d250df-3018-433a-b9b2-f1a02f4ed89f", "content-length": "124", "date": "Wed, 15 May 2024 07:08:46 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "35d250df-3018-433a-b9b2-f1a02f4ed89f" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 203.372159ms sled_address = [fd00:1122:3344:101::1]:12345
7512024-05-15T07:08:46.464ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 952adea2-114c-4099-9bd9-86f261fe9687, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f2afc369-fa05-4e43-9c24-aaf26564f459, kind: Service { id: 952adea2-114c-4099-9bd9-86f261fe9687 }, name: Name("ntp-952adea2-114c-4099-9bd9-86f261fe9687"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 18])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } } } }, OmicronZoneConfig { id: c7d123b0-1fe2-4336-ad94-79216ae10c14, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9035da45-fa42-447f-985d-89ad334749a3, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } } } }, OmicronZoneConfig { id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
7522024-05-15T07:08:46.465ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
7532024-05-15T07:08:46.465ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7542024-05-15T07:08:46.484ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
7552024-05-15T07:08:46.484ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
7562024-05-15T07:08:46.503ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
7572024-05-15T07:08:46.503ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7582024-05-15T07:08:46.521ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7592024-05-15T07:08:46.521ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7602024-05-15T07:08:46.540ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7612024-05-15T07:08:46.540ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7622024-05-15T07:08:46.558ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7632024-05-15T07:08:46.558ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7642024-05-15T07:08:46.577ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7652024-05-15T07:08:46.577ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7662024-05-15T07:08:46.595ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7672024-05-15T07:08:46.595ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7682024-05-15T07:08:46.613ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
7692024-05-15T07:08:46.613ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
7702024-05-15T07:08:46.614ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
7712024-05-15T07:08:46.614ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
7722024-05-15T07:08:46.648ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
7732024-05-15T07:08:46.648ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
7742024-05-15T07:08:46.648ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
7752024-05-15T07:08:46.648ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
7762024-05-15T07:08:46.648ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
7772024-05-15T07:08:46.648ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9", TimeNotSynchronized), ("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14", TimeNotSynchronized), ("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364", TimeNotSynchronized), ("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015", TimeNotSynchronized), ("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 183095 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46219 req_id = cdd554ae-a04d-4415-980e-8967950ddb2b response_code = 500 uri = /omicron-zones
7782024-05-15T07:08:46.648ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "cdd554ae-a04d-4415-980e-8967950ddb2b", "content-length": "124", "date": "Wed, 15 May 2024 07:08:46 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "cdd554ae-a04d-4415-980e-8967950ddb2b" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 115.118132ms sled_address = [fd00:1122:3344:101::1]:12345
7792024-05-15T07:08:46.764ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 952adea2-114c-4099-9bd9-86f261fe9687, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f2afc369-fa05-4e43-9c24-aaf26564f459, kind: Service { id: 952adea2-114c-4099-9bd9-86f261fe9687 }, name: Name("ntp-952adea2-114c-4099-9bd9-86f261fe9687"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 18])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } } } }, OmicronZoneConfig { id: c7d123b0-1fe2-4336-ad94-79216ae10c14, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9035da45-fa42-447f-985d-89ad334749a3, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } } } }, OmicronZoneConfig { id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
7802024-05-15T07:08:46.764ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
7812024-05-15T07:08:46.765ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7822024-05-15T07:08:46.783ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
7832024-05-15T07:08:46.783ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
7842024-05-15T07:08:46.802ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
7852024-05-15T07:08:46.802ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7862024-05-15T07:08:46.821ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7872024-05-15T07:08:46.821ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7882024-05-15T07:08:46.840ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7892024-05-15T07:08:46.840ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7902024-05-15T07:08:46.859ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7912024-05-15T07:08:46.859ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7922024-05-15T07:08:46.876ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7932024-05-15T07:08:46.876ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7942024-05-15T07:08:46.893ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7952024-05-15T07:08:46.893ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7962024-05-15T07:08:46.910ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
7972024-05-15T07:08:46.910ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
7982024-05-15T07:08:46.910ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
7992024-05-15T07:08:46.910ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
8002024-05-15T07:08:46.943ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
8012024-05-15T07:08:46.943ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
8022024-05-15T07:08:46.943ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
8032024-05-15T07:08:46.943ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
8042024-05-15T07:08:46.943ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
8052024-05-15T07:08:46.943ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9", TimeNotSynchronized), ("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3", TimeNotSynchronized), ("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015", TimeNotSynchronized), ("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364", TimeNotSynchronized), ("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 179057 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46219 req_id = 09293551-9be4-411a-9b90-c6e229a601c3 response_code = 500 uri = /omicron-zones
8062024-05-15T07:08:46.944ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "09293551-9be4-411a-9b90-c6e229a601c3", "content-length": "124", "date": "Wed, 15 May 2024 07:08:46 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "09293551-9be4-411a-9b90-c6e229a601c3" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 303.627773ms sled_address = [fd00:1122:3344:101::1]:12345
8072024-05-15T07:08:47.249ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 952adea2-114c-4099-9bd9-86f261fe9687, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f2afc369-fa05-4e43-9c24-aaf26564f459, kind: Service { id: 952adea2-114c-4099-9bd9-86f261fe9687 }, name: Name("ntp-952adea2-114c-4099-9bd9-86f261fe9687"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 18])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } } } }, OmicronZoneConfig { id: c7d123b0-1fe2-4336-ad94-79216ae10c14, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9035da45-fa42-447f-985d-89ad334749a3, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } } } }, OmicronZoneConfig { id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
8082024-05-15T07:08:47.249ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8092024-05-15T07:08:47.249ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8102024-05-15T07:08:47.268ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8112024-05-15T07:08:47.268ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
8122024-05-15T07:08:47.288ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8132024-05-15T07:08:47.288ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8142024-05-15T07:08:47.306ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8152024-05-15T07:08:47.306ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8162024-05-15T07:08:47.325ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8172024-05-15T07:08:47.325ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8182024-05-15T07:08:47.344ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8192024-05-15T07:08:47.344ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8202024-05-15T07:08:47.363ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8212024-05-15T07:08:47.363ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8222024-05-15T07:08:47.382ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8232024-05-15T07:08:47.382ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8242024-05-15T07:08:48.002ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
8252024-05-15T07:08:48.002ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
8262024-05-15T07:08:48.002ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
8272024-05-15T07:08:48.002ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
8282024-05-15T07:08:48.036ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
8292024-05-15T07:08:48.036ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
8302024-05-15T07:08:48.036ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
8312024-05-15T07:08:48.036ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
8322024-05-15T07:08:48.036ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
8332024-05-15T07:08:48.036ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14", TimeNotSynchronized), ("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3", TimeNotSynchronized), ("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015", TimeNotSynchronized), ("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364", TimeNotSynchronized), ("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 184478 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46219 req_id = 309cae2b-f25a-4047-bf88-9cd130250999 response_code = 500 uri = /omicron-zones
8342024-05-15T07:08:48.036ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "309cae2b-f25a-4047-bf88-9cd130250999", "content-length": "124", "date": "Wed, 15 May 2024 07:08:48 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "309cae2b-f25a-4047-bf88-9cd130250999" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 317.193025ms sled_address = [fd00:1122:3344:101::1]:12345
8352024-05-15T07:08:48.353ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 952adea2-114c-4099-9bd9-86f261fe9687, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f2afc369-fa05-4e43-9c24-aaf26564f459, kind: Service { id: 952adea2-114c-4099-9bd9-86f261fe9687 }, name: Name("ntp-952adea2-114c-4099-9bd9-86f261fe9687"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 18])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } } } }, OmicronZoneConfig { id: c7d123b0-1fe2-4336-ad94-79216ae10c14, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9035da45-fa42-447f-985d-89ad334749a3, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } } } }, OmicronZoneConfig { id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
8362024-05-15T07:08:48.354ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8372024-05-15T07:08:48.354ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8382024-05-15T07:08:48.372ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8392024-05-15T07:08:48.372ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
8402024-05-15T07:08:48.391ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8412024-05-15T07:08:48.391ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8422024-05-15T07:08:48.409ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8432024-05-15T07:08:48.409ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8442024-05-15T07:08:48.428ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8452024-05-15T07:08:48.428ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8462024-05-15T07:08:48.447ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8472024-05-15T07:08:48.447ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8482024-05-15T07:08:48.465ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8492024-05-15T07:08:48.465ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8502024-05-15T07:08:48.483ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8512024-05-15T07:08:48.483ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8522024-05-15T07:08:48.501ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
8532024-05-15T07:08:48.501ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
8542024-05-15T07:08:48.501ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
8552024-05-15T07:08:48.502ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
8562024-05-15T07:08:48.533ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
8572024-05-15T07:08:48.533ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
8582024-05-15T07:08:48.533ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
8592024-05-15T07:08:48.534ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
8602024-05-15T07:08:48.534ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
8612024-05-15T07:08:48.534ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015", TimeNotSynchronized), ("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14", TimeNotSynchronized), ("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3", TimeNotSynchronized), ("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9", TimeNotSynchronized), ("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 180104 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46219 req_id = e765f94e-f27f-4e1f-9865-e5413cef99c2 response_code = 500 uri = /omicron-zones
8622024-05-15T07:08:48.534ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "e765f94e-f27f-4e1f-9865-e5413cef99c2", "content-length": "124", "date": "Wed, 15 May 2024 07:08:48 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "e765f94e-f27f-4e1f-9865-e5413cef99c2" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 162.071826ms sled_address = [fd00:1122:3344:101::1]:12345
8632024-05-15T07:08:48.697ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 952adea2-114c-4099-9bd9-86f261fe9687, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f2afc369-fa05-4e43-9c24-aaf26564f459, kind: Service { id: 952adea2-114c-4099-9bd9-86f261fe9687 }, name: Name("ntp-952adea2-114c-4099-9bd9-86f261fe9687"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 18])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } } } }, OmicronZoneConfig { id: c7d123b0-1fe2-4336-ad94-79216ae10c14, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9035da45-fa42-447f-985d-89ad334749a3, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } } } }, OmicronZoneConfig { id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
8642024-05-15T07:08:48.698ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8652024-05-15T07:08:48.698ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8662024-05-15T07:08:48.716ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8672024-05-15T07:08:48.716ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
8682024-05-15T07:08:48.734ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8692024-05-15T07:08:48.735ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8702024-05-15T07:08:48.753ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8712024-05-15T07:08:48.753ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8722024-05-15T07:08:48.770ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8732024-05-15T07:08:48.770ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8742024-05-15T07:08:48.788ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8752024-05-15T07:08:48.788ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8762024-05-15T07:08:48.805ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8772024-05-15T07:08:48.805ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8782024-05-15T07:08:48.823ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8792024-05-15T07:08:48.823ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8802024-05-15T07:08:48.840ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
8812024-05-15T07:08:48.840ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
8822024-05-15T07:08:48.841ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
8832024-05-15T07:08:48.841ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
8842024-05-15T07:08:48.873ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
8852024-05-15T07:08:48.873ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
8862024-05-15T07:08:48.873ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
8872024-05-15T07:08:48.873ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
8882024-05-15T07:08:48.873ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
8892024-05-15T07:08:48.873ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015", TimeNotSynchronized), ("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9", TimeNotSynchronized), ("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14", TimeNotSynchronized), ("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3", TimeNotSynchronized), ("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 175780 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46219 req_id = 2e2c06bc-a7c9-4461-b5d9-5eed21da466e response_code = 500 uri = /omicron-zones
8902024-05-15T07:08:48.873ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "2e2c06bc-a7c9-4461-b5d9-5eed21da466e", "content-length": "124", "date": "Wed, 15 May 2024 07:08:48 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "2e2c06bc-a7c9-4461-b5d9-5eed21da466e" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 496.32777ms sled_address = [fd00:1122:3344:101::1]:12345
8912024-05-15T07:08:49.371ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 952adea2-114c-4099-9bd9-86f261fe9687, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f2afc369-fa05-4e43-9c24-aaf26564f459, kind: Service { id: 952adea2-114c-4099-9bd9-86f261fe9687 }, name: Name("ntp-952adea2-114c-4099-9bd9-86f261fe9687"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 18])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } } } }, OmicronZoneConfig { id: c7d123b0-1fe2-4336-ad94-79216ae10c14, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9035da45-fa42-447f-985d-89ad334749a3, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } } } }, OmicronZoneConfig { id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
8922024-05-15T07:08:49.371ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8932024-05-15T07:08:49.371ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8942024-05-15T07:08:49.390ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8952024-05-15T07:08:49.390ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
8962024-05-15T07:08:49.409ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8972024-05-15T07:08:49.409ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8982024-05-15T07:08:49.429ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8992024-05-15T07:08:49.429ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9002024-05-15T07:08:49.448ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9012024-05-15T07:08:49.448ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9022024-05-15T07:08:49.467ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9032024-05-15T07:08:49.467ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9042024-05-15T07:08:49.486ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9052024-05-15T07:08:49.486ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9062024-05-15T07:08:49.503ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9072024-05-15T07:08:49.503ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9082024-05-15T07:08:49.520ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
9092024-05-15T07:08:49.520ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
9102024-05-15T07:08:49.520ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
9112024-05-15T07:08:49.520ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
9122024-05-15T07:08:49.552ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
9132024-05-15T07:08:49.552ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
9142024-05-15T07:08:49.552ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
9152024-05-15T07:08:49.552ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
9162024-05-15T07:08:49.552ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
9172024-05-15T07:08:49.552ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14", TimeNotSynchronized), ("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3", TimeNotSynchronized), ("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015", TimeNotSynchronized), ("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9", TimeNotSynchronized), ("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 180737 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46219 req_id = c4fe0d09-b3cf-4eb7-a452-510c355c4bdb response_code = 500 uri = /omicron-zones
9182024-05-15T07:08:49.552ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "c4fe0d09-b3cf-4eb7-a452-510c355c4bdb", "content-length": "124", "date": "Wed, 15 May 2024 07:08:49 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "c4fe0d09-b3cf-4eb7-a452-510c355c4bdb" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 583.687742ms sled_address = [fd00:1122:3344:101::1]:12345
9192024-05-15T07:08:50.136ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 952adea2-114c-4099-9bd9-86f261fe9687, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f2afc369-fa05-4e43-9c24-aaf26564f459, kind: Service { id: 952adea2-114c-4099-9bd9-86f261fe9687 }, name: Name("ntp-952adea2-114c-4099-9bd9-86f261fe9687"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 18])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } } } }, OmicronZoneConfig { id: c7d123b0-1fe2-4336-ad94-79216ae10c14, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9035da45-fa42-447f-985d-89ad334749a3, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } } } }, OmicronZoneConfig { id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
9202024-05-15T07:08:50.137ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9212024-05-15T07:08:50.137ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9222024-05-15T07:08:50.155ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9232024-05-15T07:08:50.155ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
9242024-05-15T07:08:50.172ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9252024-05-15T07:08:50.172ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9262024-05-15T07:08:50.191ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9272024-05-15T07:08:50.191ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9282024-05-15T07:08:50.209ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9292024-05-15T07:08:50.209ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9302024-05-15T07:08:50.228ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9312024-05-15T07:08:50.228ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9322024-05-15T07:08:50.245ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9332024-05-15T07:08:50.245ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9342024-05-15T07:08:50.262ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9352024-05-15T07:08:50.262ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9362024-05-15T07:08:50.280ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
9372024-05-15T07:08:50.280ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
9382024-05-15T07:08:50.280ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
9392024-05-15T07:08:50.280ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
9402024-05-15T07:08:50.313ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
9412024-05-15T07:08:50.313ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
9422024-05-15T07:08:50.313ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
9432024-05-15T07:08:50.313ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
9442024-05-15T07:08:50.313ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
9452024-05-15T07:08:50.313ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3", TimeNotSynchronized), ("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364", TimeNotSynchronized), ("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015", TimeNotSynchronized), ("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9", TimeNotSynchronized), ("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 176555 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46219 req_id = a0ef122c-7a62-43ed-88c5-a4c4a78a5811 response_code = 500 uri = /omicron-zones
9462024-05-15T07:08:50.313ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "a0ef122c-7a62-43ed-88c5-a4c4a78a5811", "content-length": "124", "date": "Wed, 15 May 2024 07:08:49 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "a0ef122c-7a62-43ed-88c5-a4c4a78a5811" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 494.119885ms sled_address = [fd00:1122:3344:101::1]:12345
9472024-05-15T07:08:50.808ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 952adea2-114c-4099-9bd9-86f261fe9687, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f2afc369-fa05-4e43-9c24-aaf26564f459, kind: Service { id: 952adea2-114c-4099-9bd9-86f261fe9687 }, name: Name("ntp-952adea2-114c-4099-9bd9-86f261fe9687"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 18])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } } } }, OmicronZoneConfig { id: c7d123b0-1fe2-4336-ad94-79216ae10c14, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9035da45-fa42-447f-985d-89ad334749a3, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } } } }, OmicronZoneConfig { id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
9482024-05-15T07:08:50.809ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9492024-05-15T07:08:50.809ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9502024-05-15T07:08:50.828ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9512024-05-15T07:08:50.828ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
9522024-05-15T07:08:50.846ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9532024-05-15T07:08:50.846ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9542024-05-15T07:08:50.866ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9552024-05-15T07:08:50.866ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9562024-05-15T07:08:50.883ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9572024-05-15T07:08:50.883ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9582024-05-15T07:08:50.902ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9592024-05-15T07:08:50.902ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9602024-05-15T07:08:50.919ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9612024-05-15T07:08:50.919ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9622024-05-15T07:08:50.937ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9632024-05-15T07:08:50.937ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9642024-05-15T07:08:50.955ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
9652024-05-15T07:08:50.955ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
9662024-05-15T07:08:50.956ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
9672024-05-15T07:08:50.956ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
9682024-05-15T07:08:50.988ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
9692024-05-15T07:08:50.988ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
9702024-05-15T07:08:50.988ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
9712024-05-15T07:08:50.988ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
9722024-05-15T07:08:50.988ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
9732024-05-15T07:08:50.988ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3", TimeNotSynchronized), ("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015", TimeNotSynchronized), ("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9", TimeNotSynchronized), ("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14", TimeNotSynchronized), ("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 179716 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46219 req_id = bee4f337-00de-4ea4-bfa1-e2ba2a9dc547 response_code = 500 uri = /omicron-zones
9742024-05-15T07:08:50.988ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "bee4f337-00de-4ea4-bfa1-e2ba2a9dc547", "content-length": "124", "date": "Wed, 15 May 2024 07:08:50 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "bee4f337-00de-4ea4-bfa1-e2ba2a9dc547" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 622.18083ms sled_address = [fd00:1122:3344:101::1]:12345
9752024-05-15T07:08:51.612ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 952adea2-114c-4099-9bd9-86f261fe9687, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f2afc369-fa05-4e43-9c24-aaf26564f459, kind: Service { id: 952adea2-114c-4099-9bd9-86f261fe9687 }, name: Name("ntp-952adea2-114c-4099-9bd9-86f261fe9687"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 18])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } } } }, OmicronZoneConfig { id: c7d123b0-1fe2-4336-ad94-79216ae10c14, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9035da45-fa42-447f-985d-89ad334749a3, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } } } }, OmicronZoneConfig { id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
9762024-05-15T07:08:51.612ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9772024-05-15T07:08:51.612ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9782024-05-15T07:08:51.633ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9792024-05-15T07:08:51.633ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
9802024-05-15T07:08:51.653ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9812024-05-15T07:08:51.653ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9822024-05-15T07:08:51.672ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9832024-05-15T07:08:51.672ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9842024-05-15T07:08:51.694ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9852024-05-15T07:08:51.694ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9862024-05-15T07:08:51.714ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9872024-05-15T07:08:51.714ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9882024-05-15T07:08:51.732ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9892024-05-15T07:08:51.732ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9902024-05-15T07:08:51.751ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9912024-05-15T07:08:51.751ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9922024-05-15T07:08:51.771ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
9932024-05-15T07:08:51.771ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
9942024-05-15T07:08:51.771ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
9952024-05-15T07:08:51.771ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
9962024-05-15T07:08:51.805ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
9972024-05-15T07:08:51.805ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
9982024-05-15T07:08:51.805ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
9992024-05-15T07:08:51.805ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
10002024-05-15T07:08:51.805ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
10012024-05-15T07:08:51.805ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9", TimeNotSynchronized), ("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364", TimeNotSynchronized), ("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015", TimeNotSynchronized), ("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14", TimeNotSynchronized), ("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 193369 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46219 req_id = f9479879-cc41-4b37-8658-dfcce586acd4 response_code = 500 uri = /omicron-zones
10022024-05-15T07:08:51.805ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "f9479879-cc41-4b37-8658-dfcce586acd4", "content-length": "124", "date": "Wed, 15 May 2024 07:08:50 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "f9479879-cc41-4b37-8658-dfcce586acd4" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 802.125606ms sled_address = [fd00:1122:3344:101::1]:12345
10032024-05-15T07:08:52.609ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 952adea2-114c-4099-9bd9-86f261fe9687, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f2afc369-fa05-4e43-9c24-aaf26564f459, kind: Service { id: 952adea2-114c-4099-9bd9-86f261fe9687 }, name: Name("ntp-952adea2-114c-4099-9bd9-86f261fe9687"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 18])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } } } }, OmicronZoneConfig { id: c7d123b0-1fe2-4336-ad94-79216ae10c14, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9035da45-fa42-447f-985d-89ad334749a3, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } } } }, OmicronZoneConfig { id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
10042024-05-15T07:08:52.609ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10052024-05-15T07:08:52.609ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10062024-05-15T07:08:52.629ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10072024-05-15T07:08:52.629ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
10082024-05-15T07:08:52.648ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10092024-05-15T07:08:52.648ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10102024-05-15T07:08:52.667ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10112024-05-15T07:08:52.667ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10122024-05-15T07:08:52.686ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10132024-05-15T07:08:52.686ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10142024-05-15T07:08:52.705ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10152024-05-15T07:08:52.705ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10162024-05-15T07:08:52.724ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10172024-05-15T07:08:52.724ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10182024-05-15T07:08:52.743ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10192024-05-15T07:08:52.743ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10202024-05-15T07:08:52.762ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
10212024-05-15T07:08:52.762ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
10222024-05-15T07:08:52.762ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
10232024-05-15T07:08:52.762ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
10242024-05-15T07:08:52.796ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
10252024-05-15T07:08:52.796ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
10262024-05-15T07:08:52.796ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
10272024-05-15T07:08:52.796ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
10282024-05-15T07:08:52.796ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
10292024-05-15T07:08:52.796ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14", TimeNotSynchronized), ("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9", TimeNotSynchronized), ("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3", TimeNotSynchronized), ("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015", TimeNotSynchronized), ("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 186985 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46219 req_id = 30c7abaf-6fbc-49a3-9280-e2d1ac8e5cf8 response_code = 500 uri = /omicron-zones
10302024-05-15T07:08:52.796ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "30c7abaf-6fbc-49a3-9280-e2d1ac8e5cf8", "content-length": "124", "date": "Wed, 15 May 2024 07:08:52 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "30c7abaf-6fbc-49a3-9280-e2d1ac8e5cf8" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 720.347015ms sled_address = [fd00:1122:3344:101::1]:12345
10312024-05-15T07:08:53.518ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 952adea2-114c-4099-9bd9-86f261fe9687, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f2afc369-fa05-4e43-9c24-aaf26564f459, kind: Service { id: 952adea2-114c-4099-9bd9-86f261fe9687 }, name: Name("ntp-952adea2-114c-4099-9bd9-86f261fe9687"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 18])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } } } }, OmicronZoneConfig { id: c7d123b0-1fe2-4336-ad94-79216ae10c14, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9035da45-fa42-447f-985d-89ad334749a3, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } } } }, OmicronZoneConfig { id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
10322024-05-15T07:08:53.519ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10332024-05-15T07:08:53.519ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10342024-05-15T07:08:53.538ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10352024-05-15T07:08:53.538ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
10362024-05-15T07:08:53.558ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10372024-05-15T07:08:53.558ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10382024-05-15T07:08:53.576ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10392024-05-15T07:08:53.576ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10402024-05-15T07:08:53.595ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10412024-05-15T07:08:53.595ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10422024-05-15T07:08:53.614ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10432024-05-15T07:08:53.614ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10442024-05-15T07:08:53.632ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10452024-05-15T07:08:53.632ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10462024-05-15T07:08:53.649ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10472024-05-15T07:08:53.649ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10482024-05-15T07:08:53.668ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
10492024-05-15T07:08:53.668ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
10502024-05-15T07:08:53.668ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
10512024-05-15T07:08:53.668ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
10522024-05-15T07:08:53.700ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
10532024-05-15T07:08:53.700ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
10542024-05-15T07:08:53.700ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
10552024-05-15T07:08:53.700ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
10562024-05-15T07:08:53.700ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
10572024-05-15T07:08:53.700ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015", TimeNotSynchronized), ("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9", TimeNotSynchronized), ("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14", TimeNotSynchronized), ("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3", TimeNotSynchronized), ("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 182559 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46219 req_id = f810d3be-09e4-4edd-8da5-f9f4f8ad5a2d response_code = 500 uri = /omicron-zones
10582024-05-15T07:08:53.701ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "f810d3be-09e4-4edd-8da5-f9f4f8ad5a2d", "content-length": "124", "date": "Wed, 15 May 2024 07:08:53 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "f810d3be-09e4-4edd-8da5-f9f4f8ad5a2d" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 981.080084ms sled_address = [fd00:1122:3344:101::1]:12345
10592024-05-15T07:08:54.683ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 952adea2-114c-4099-9bd9-86f261fe9687, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f2afc369-fa05-4e43-9c24-aaf26564f459, kind: Service { id: 952adea2-114c-4099-9bd9-86f261fe9687 }, name: Name("ntp-952adea2-114c-4099-9bd9-86f261fe9687"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 18])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } } } }, OmicronZoneConfig { id: c7d123b0-1fe2-4336-ad94-79216ae10c14, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9035da45-fa42-447f-985d-89ad334749a3, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } } } }, OmicronZoneConfig { id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
10602024-05-15T07:08:54.684ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10612024-05-15T07:08:54.684ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10622024-05-15T07:08:54.704ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10632024-05-15T07:08:54.704ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
10642024-05-15T07:08:54.724ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10652024-05-15T07:08:54.724ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10662024-05-15T07:08:54.743ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10672024-05-15T07:08:54.743ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10682024-05-15T07:08:54.763ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10692024-05-15T07:08:54.763ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10702024-05-15T07:08:54.782ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10712024-05-15T07:08:54.782ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10722024-05-15T07:08:54.801ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10732024-05-15T07:08:54.801ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10742024-05-15T07:08:54.820ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10752024-05-15T07:08:54.820ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10762024-05-15T07:08:54.839ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
10772024-05-15T07:08:54.839ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
10782024-05-15T07:08:54.839ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
10792024-05-15T07:08:54.839ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
10802024-05-15T07:08:54.873ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
10812024-05-15T07:08:54.873ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
10822024-05-15T07:08:54.873ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
10832024-05-15T07:08:54.873ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
10842024-05-15T07:08:54.873ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
10852024-05-15T07:08:54.873ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14", TimeNotSynchronized), ("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9", TimeNotSynchronized), ("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364", TimeNotSynchronized), ("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3", TimeNotSynchronized), ("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 189556 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46219 req_id = be5c5b64-aaf1-4b3a-a770-0cd86853563f response_code = 500 uri = /omicron-zones
10862024-05-15T07:08:54.873ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "be5c5b64-aaf1-4b3a-a770-0cd86853563f", "content-length": "124", "date": "Wed, 15 May 2024 07:08:54 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "be5c5b64-aaf1-4b3a-a770-0cd86853563f" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 1.679257772s sled_address = [fd00:1122:3344:101::1]:12345
10872024-05-15T07:08:56.341ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 166.451793ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:426 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
10882024-05-15T07:08:56.554ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 952adea2-114c-4099-9bd9-86f261fe9687, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f2afc369-fa05-4e43-9c24-aaf26564f459, kind: Service { id: 952adea2-114c-4099-9bd9-86f261fe9687 }, name: Name("ntp-952adea2-114c-4099-9bd9-86f261fe9687"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 18])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } } } }, OmicronZoneConfig { id: c7d123b0-1fe2-4336-ad94-79216ae10c14, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9035da45-fa42-447f-985d-89ad334749a3, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } } } }, OmicronZoneConfig { id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
10892024-05-15T07:08:56.554ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10902024-05-15T07:08:56.554ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10912024-05-15T07:08:56.576ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10922024-05-15T07:08:56.576ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
10932024-05-15T07:08:56.595ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10942024-05-15T07:08:56.595ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10952024-05-15T07:08:56.614ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10962024-05-15T07:08:56.614ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10972024-05-15T07:08:56.633ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10982024-05-15T07:08:56.633ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10992024-05-15T07:08:56.652ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11002024-05-15T07:08:56.652ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11012024-05-15T07:08:56.671ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11022024-05-15T07:08:56.671ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11032024-05-15T07:08:56.689ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11042024-05-15T07:08:56.689ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11052024-05-15T07:08:56.708ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
11062024-05-15T07:08:56.708ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
11072024-05-15T07:08:56.709ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
11082024-05-15T07:08:56.709ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
11092024-05-15T07:08:56.743ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
11102024-05-15T07:08:56.743ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
11112024-05-15T07:08:56.743ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
11122024-05-15T07:08:56.743ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
11132024-05-15T07:08:56.743ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
11142024-05-15T07:08:56.743ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015", TimeNotSynchronized), ("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14", TimeNotSynchronized), ("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3", TimeNotSynchronized), ("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364", TimeNotSynchronized), ("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 188960 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46219 req_id = 6d962ca5-9034-422f-a2b2-e036abeb0d71 response_code = 500 uri = /omicron-zones
11152024-05-15T07:08:56.743ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "6d962ca5-9034-422f-a2b2-e036abeb0d71", "content-length": "124", "date": "Wed, 15 May 2024 07:08:56 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "6d962ca5-9034-422f-a2b2-e036abeb0d71" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 1.228987941s sled_address = [fd00:1122:3344:101::1]:12345
11162024-05-15T07:08:57.973ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 952adea2-114c-4099-9bd9-86f261fe9687, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f2afc369-fa05-4e43-9c24-aaf26564f459, kind: Service { id: 952adea2-114c-4099-9bd9-86f261fe9687 }, name: Name("ntp-952adea2-114c-4099-9bd9-86f261fe9687"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 18])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } } } }, OmicronZoneConfig { id: c7d123b0-1fe2-4336-ad94-79216ae10c14, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9035da45-fa42-447f-985d-89ad334749a3, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } } } }, OmicronZoneConfig { id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
11172024-05-15T07:08:57.974ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11182024-05-15T07:08:57.974ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11192024-05-15T07:08:57.995ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11202024-05-15T07:08:57.995ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
11212024-05-15T07:08:58.016ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11222024-05-15T07:08:58.016ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11232024-05-15T07:08:58.035ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11242024-05-15T07:08:58.035ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11252024-05-15T07:08:58.055ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11262024-05-15T07:08:58.055ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11272024-05-15T07:08:58.074ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11282024-05-15T07:08:58.074ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11292024-05-15T07:08:58.093ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11302024-05-15T07:08:58.093ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11312024-05-15T07:08:58.113ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11322024-05-15T07:08:58.113ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11332024-05-15T07:08:58.132ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
11342024-05-15T07:08:58.132ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
11352024-05-15T07:08:58.132ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
11362024-05-15T07:08:58.132ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
11372024-05-15T07:08:58.166ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
11382024-05-15T07:08:58.166ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
11392024-05-15T07:08:58.166ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
11402024-05-15T07:08:58.166ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
11412024-05-15T07:08:58.166ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
11422024-05-15T07:08:58.166ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14", TimeNotSynchronized), ("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015", TimeNotSynchronized), ("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9", TimeNotSynchronized), ("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3", TimeNotSynchronized), ("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 193379 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46219 req_id = c19a2e72-69e2-4673-82f6-299687639c71 response_code = 500 uri = /omicron-zones
11432024-05-15T07:08:58.166ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "c19a2e72-69e2-4673-82f6-299687639c71", "content-length": "124", "date": "Wed, 15 May 2024 07:08:58 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "c19a2e72-69e2-4673-82f6-299687639c71" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 2.594466711s sled_address = [fd00:1122:3344:101::1]:12345
11442024-05-15T07:09:00.762ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 952adea2-114c-4099-9bd9-86f261fe9687, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f2afc369-fa05-4e43-9c24-aaf26564f459, kind: Service { id: 952adea2-114c-4099-9bd9-86f261fe9687 }, name: Name("ntp-952adea2-114c-4099-9bd9-86f261fe9687"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 18])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } } } }, OmicronZoneConfig { id: c7d123b0-1fe2-4336-ad94-79216ae10c14, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9035da45-fa42-447f-985d-89ad334749a3, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } } } }, OmicronZoneConfig { id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
11452024-05-15T07:09:00.763ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11462024-05-15T07:09:00.763ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11472024-05-15T07:09:00.783ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11482024-05-15T07:09:00.783ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
11492024-05-15T07:09:00.803ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11502024-05-15T07:09:00.803ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11512024-05-15T07:09:00.821ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11522024-05-15T07:09:00.821ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11532024-05-15T07:09:00.842ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11542024-05-15T07:09:00.842ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11552024-05-15T07:09:00.860ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11562024-05-15T07:09:00.860ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11572024-05-15T07:09:00.879ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11582024-05-15T07:09:00.879ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11592024-05-15T07:09:00.898ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11602024-05-15T07:09:00.898ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11612024-05-15T07:09:00.917ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
11622024-05-15T07:09:00.917ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
11632024-05-15T07:09:00.917ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
11642024-05-15T07:09:00.917ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
11652024-05-15T07:09:00.950ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
11662024-05-15T07:09:00.950ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
11672024-05-15T07:09:00.950ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
11682024-05-15T07:09:00.950ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
11692024-05-15T07:09:00.950ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
11702024-05-15T07:09:00.950ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3", TimeNotSynchronized), ("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015", TimeNotSynchronized), ("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9", TimeNotSynchronized), ("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364", TimeNotSynchronized), ("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 187276 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46219 req_id = 11266bbc-2630-4db7-adf9-bced0d5687ee response_code = 500 uri = /omicron-zones
11712024-05-15T07:09:00.950ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "11266bbc-2630-4db7-adf9-bced0d5687ee", "content-length": "124", "date": "Wed, 15 May 2024 07:09:00 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "11266bbc-2630-4db7-adf9-bced0d5687ee" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 2.481282364s sled_address = [fd00:1122:3344:101::1]:12345
11722024-05-15T07:09:03.209ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:47579
11732024-05-15T07:09:03.209ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 72 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:47579 req_id = d0393f44-c042-4cef-93a6-09985ad997e4 response_code = 200 uri = /baseboard
11742024-05-15T07:09:03.432ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 952adea2-114c-4099-9bd9-86f261fe9687, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f2afc369-fa05-4e43-9c24-aaf26564f459, kind: Service { id: 952adea2-114c-4099-9bd9-86f261fe9687 }, name: Name("ntp-952adea2-114c-4099-9bd9-86f261fe9687"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 18])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } } } }, OmicronZoneConfig { id: c7d123b0-1fe2-4336-ad94-79216ae10c14, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9035da45-fa42-447f-985d-89ad334749a3, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } } } }, OmicronZoneConfig { id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
11752024-05-15T07:09:03.432ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11762024-05-15T07:09:03.432ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11772024-05-15T07:09:03.453ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11782024-05-15T07:09:03.453ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
11792024-05-15T07:09:03.473ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11802024-05-15T07:09:03.473ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11812024-05-15T07:09:03.493ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11822024-05-15T07:09:03.493ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11832024-05-15T07:09:03.513ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11842024-05-15T07:09:03.513ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11852024-05-15T07:09:03.533ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11862024-05-15T07:09:03.533ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11872024-05-15T07:09:03.551ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11882024-05-15T07:09:03.551ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11892024-05-15T07:09:03.571ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11902024-05-15T07:09:03.571ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11912024-05-15T07:09:03.590ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
11922024-05-15T07:09:03.590ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
11932024-05-15T07:09:03.590ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
11942024-05-15T07:09:03.590ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
11952024-05-15T07:09:03.624ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
11962024-05-15T07:09:03.624ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
11972024-05-15T07:09:03.624ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
11982024-05-15T07:09:03.624ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
11992024-05-15T07:09:03.624ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
12002024-05-15T07:09:03.624ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9", TimeNotSynchronized), ("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364", TimeNotSynchronized), ("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015", TimeNotSynchronized), ("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14", TimeNotSynchronized), ("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 192497 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46219 req_id = 6632bf65-1d4d-4339-a9cc-462c12182c40 response_code = 500 uri = /omicron-zones
12012024-05-15T07:09:03.625ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "6632bf65-1d4d-4339-a9cc-462c12182c40", "content-length": "124", "date": "Wed, 15 May 2024 07:09:03 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "6632bf65-1d4d-4339-a9cc-462c12182c40" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 3.096175391s sled_address = [fd00:1122:3344:101::1]:12345
12022024-05-15T07:09:06.722ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 952adea2-114c-4099-9bd9-86f261fe9687, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f2afc369-fa05-4e43-9c24-aaf26564f459, kind: Service { id: 952adea2-114c-4099-9bd9-86f261fe9687 }, name: Name("ntp-952adea2-114c-4099-9bd9-86f261fe9687"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 18])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } } } }, OmicronZoneConfig { id: c7d123b0-1fe2-4336-ad94-79216ae10c14, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9035da45-fa42-447f-985d-89ad334749a3, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } } } }, OmicronZoneConfig { id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
12032024-05-15T07:09:06.722ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12042024-05-15T07:09:06.722ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12052024-05-15T07:09:06.745ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12062024-05-15T07:09:06.745ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
12072024-05-15T07:09:06.765ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12082024-05-15T07:09:06.765ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12092024-05-15T07:09:06.784ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12102024-05-15T07:09:06.784ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12112024-05-15T07:09:06.804ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12122024-05-15T07:09:06.804ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12132024-05-15T07:09:06.823ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12142024-05-15T07:09:06.823ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12152024-05-15T07:09:06.841ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12162024-05-15T07:09:06.841ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12172024-05-15T07:09:06.861ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12182024-05-15T07:09:06.861ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12192024-05-15T07:09:06.880ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
12202024-05-15T07:09:06.880ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
12212024-05-15T07:09:06.880ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
12222024-05-15T07:09:06.881ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
12232024-05-15T07:09:06.916ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
12242024-05-15T07:09:06.916ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
12252024-05-15T07:09:06.916ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
12262024-05-15T07:09:06.916ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
12272024-05-15T07:09:06.916ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
12282024-05-15T07:09:06.916ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3", TimeNotSynchronized), ("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015", TimeNotSynchronized), ("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9", TimeNotSynchronized), ("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14", TimeNotSynchronized), ("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 194219 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46219 req_id = 50753cb7-ee78-4b91-8bce-132c0612f5a8 response_code = 500 uri = /omicron-zones
12292024-05-15T07:09:06.917ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "50753cb7-ee78-4b91-8bce-132c0612f5a8", "content-length": "124", "date": "Wed, 15 May 2024 07:09:06 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "50753cb7-ee78-4b91-8bce-132c0612f5a8" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 3.294517183s sled_address = [fd00:1122:3344:101::1]:12345
12302024-05-15T07:09:10.212ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 952adea2-114c-4099-9bd9-86f261fe9687, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f2afc369-fa05-4e43-9c24-aaf26564f459, kind: Service { id: 952adea2-114c-4099-9bd9-86f261fe9687 }, name: Name("ntp-952adea2-114c-4099-9bd9-86f261fe9687"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 18])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } } } }, OmicronZoneConfig { id: c7d123b0-1fe2-4336-ad94-79216ae10c14, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9035da45-fa42-447f-985d-89ad334749a3, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } } } }, OmicronZoneConfig { id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
12312024-05-15T07:09:10.212ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12322024-05-15T07:09:10.212ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12332024-05-15T07:09:10.233ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12342024-05-15T07:09:10.233ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
12352024-05-15T07:09:10.254ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12362024-05-15T07:09:10.254ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12372024-05-15T07:09:10.272ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12382024-05-15T07:09:10.272ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12392024-05-15T07:09:10.290ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12402024-05-15T07:09:10.290ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12412024-05-15T07:09:10.310ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12422024-05-15T07:09:10.310ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12432024-05-15T07:09:10.328ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12442024-05-15T07:09:10.328ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12452024-05-15T07:09:10.347ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12462024-05-15T07:09:10.347ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12472024-05-15T07:09:10.364ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
12482024-05-15T07:09:10.364ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
12492024-05-15T07:09:10.365ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
12502024-05-15T07:09:10.365ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
12512024-05-15T07:09:10.397ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
12522024-05-15T07:09:10.397ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
12532024-05-15T07:09:10.397ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
12542024-05-15T07:09:10.397ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
12552024-05-15T07:09:10.397ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
12562024-05-15T07:09:10.397ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9", TimeNotSynchronized), ("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015", TimeNotSynchronized), ("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3", TimeNotSynchronized), ("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14", TimeNotSynchronized), ("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 185466 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46219 req_id = badd8074-ffbb-4e77-9958-593538c0a44f response_code = 500 uri = /omicron-zones
12572024-05-15T07:09:10.398ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "badd8074-ffbb-4e77-9958-593538c0a44f", "content-length": "124", "date": "Wed, 15 May 2024 07:09:10 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "badd8074-ffbb-4e77-9958-593538c0a44f" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 3.974484717s sled_address = [fd00:1122:3344:101::1]:12345
12582024-05-15T07:09:11.542ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 654.623942ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:426 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
12592024-05-15T07:09:14.373ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 952adea2-114c-4099-9bd9-86f261fe9687, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f2afc369-fa05-4e43-9c24-aaf26564f459, kind: Service { id: 952adea2-114c-4099-9bd9-86f261fe9687 }, name: Name("ntp-952adea2-114c-4099-9bd9-86f261fe9687"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 18])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } } } }, OmicronZoneConfig { id: c7d123b0-1fe2-4336-ad94-79216ae10c14, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9035da45-fa42-447f-985d-89ad334749a3, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } } } }, OmicronZoneConfig { id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
12602024-05-15T07:09:14.374ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12612024-05-15T07:09:14.374ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12622024-05-15T07:09:14.396ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12632024-05-15T07:09:14.396ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
12642024-05-15T07:09:14.416ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12652024-05-15T07:09:14.416ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12662024-05-15T07:09:14.435ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12672024-05-15T07:09:14.435ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12682024-05-15T07:09:14.454ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12692024-05-15T07:09:14.455ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12702024-05-15T07:09:14.474ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12712024-05-15T07:09:14.474ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12722024-05-15T07:09:14.493ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12732024-05-15T07:09:14.493ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12742024-05-15T07:09:14.512ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12752024-05-15T07:09:14.512ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12762024-05-15T07:09:14.531ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
12772024-05-15T07:09:14.531ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
12782024-05-15T07:09:14.531ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
12792024-05-15T07:09:14.531ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
12802024-05-15T07:09:14.564ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
12812024-05-15T07:09:14.564ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
12822024-05-15T07:09:14.564ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
12832024-05-15T07:09:14.564ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
12842024-05-15T07:09:14.565ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
12852024-05-15T07:09:14.565ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364", TimeNotSynchronized), ("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3", TimeNotSynchronized), ("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14", TimeNotSynchronized), ("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9", TimeNotSynchronized), ("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 191798 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46219 req_id = 590619cc-fa01-4d79-91f3-43e20261f66b response_code = 500 uri = /omicron-zones
12862024-05-15T07:09:14.565ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "590619cc-fa01-4d79-91f3-43e20261f66b", "content-length": "124", "date": "Wed, 15 May 2024 07:09:14 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "590619cc-fa01-4d79-91f3-43e20261f66b" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 5.888114464s sled_address = [fd00:1122:3344:101::1]:12345
12872024-05-15T07:09:20.454ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 952adea2-114c-4099-9bd9-86f261fe9687, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f2afc369-fa05-4e43-9c24-aaf26564f459, kind: Service { id: 952adea2-114c-4099-9bd9-86f261fe9687 }, name: Name("ntp-952adea2-114c-4099-9bd9-86f261fe9687"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 18])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } } } }, OmicronZoneConfig { id: c7d123b0-1fe2-4336-ad94-79216ae10c14, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9035da45-fa42-447f-985d-89ad334749a3, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } } } }, OmicronZoneConfig { id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
12882024-05-15T07:09:20.455ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12892024-05-15T07:09:20.455ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12902024-05-15T07:09:20.477ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12912024-05-15T07:09:20.477ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
12922024-05-15T07:09:20.497ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12932024-05-15T07:09:20.497ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12942024-05-15T07:09:20.516ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12952024-05-15T07:09:20.516ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12962024-05-15T07:09:20.535ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12972024-05-15T07:09:20.535ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12982024-05-15T07:09:20.553ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12992024-05-15T07:09:20.553ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13002024-05-15T07:09:20.572ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13012024-05-15T07:09:20.572ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13022024-05-15T07:09:20.590ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13032024-05-15T07:09:20.590ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13042024-05-15T07:09:20.608ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
13052024-05-15T07:09:20.608ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
13062024-05-15T07:09:20.608ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
13072024-05-15T07:09:20.609ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
13082024-05-15T07:09:20.641ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
13092024-05-15T07:09:20.641ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
13102024-05-15T07:09:20.641ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
13112024-05-15T07:09:20.641ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
13122024-05-15T07:09:20.641ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
13132024-05-15T07:09:20.641ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9", TimeNotSynchronized), ("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364", TimeNotSynchronized), ("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015", TimeNotSynchronized), ("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14", TimeNotSynchronized), ("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 186881 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46219 req_id = dc21f78c-1282-451b-89d5-5c288e9ab8e1 response_code = 500 uri = /omicron-zones
13142024-05-15T07:09:20.641ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "dc21f78c-1282-451b-89d5-5c288e9ab8e1", "content-length": "124", "date": "Wed, 15 May 2024 07:09:20 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "dc21f78c-1282-451b-89d5-5c288e9ab8e1" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 4.32672259s sled_address = [fd00:1122:3344:101::1]:12345
13152024-05-15T07:09:24.969ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 952adea2-114c-4099-9bd9-86f261fe9687, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f2afc369-fa05-4e43-9c24-aaf26564f459, kind: Service { id: 952adea2-114c-4099-9bd9-86f261fe9687 }, name: Name("ntp-952adea2-114c-4099-9bd9-86f261fe9687"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 18])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } } } }, OmicronZoneConfig { id: c7d123b0-1fe2-4336-ad94-79216ae10c14, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9035da45-fa42-447f-985d-89ad334749a3, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } } } }, OmicronZoneConfig { id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
13162024-05-15T07:09:24.969ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13172024-05-15T07:09:24.969ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13182024-05-15T07:09:24.991ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13192024-05-15T07:09:24.991ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
13202024-05-15T07:09:25.011ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13212024-05-15T07:09:25.011ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13222024-05-15T07:09:25.030ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13232024-05-15T07:09:25.030ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13242024-05-15T07:09:25.048ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13252024-05-15T07:09:25.048ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13262024-05-15T07:09:25.068ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13272024-05-15T07:09:25.068ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13282024-05-15T07:09:25.086ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13292024-05-15T07:09:25.086ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13302024-05-15T07:09:25.106ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13312024-05-15T07:09:25.106ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13322024-05-15T07:09:25.124ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
13332024-05-15T07:09:25.124ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
13342024-05-15T07:09:25.125ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
13352024-05-15T07:09:25.125ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
13362024-05-15T07:09:25.159ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
13372024-05-15T07:09:25.159ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
13382024-05-15T07:09:25.159ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
13392024-05-15T07:09:25.159ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
13402024-05-15T07:09:25.159ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
13412024-05-15T07:09:25.159ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015", TimeNotSynchronized), ("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14", TimeNotSynchronized), ("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364", TimeNotSynchronized), ("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9", TimeNotSynchronized), ("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 190140 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46219 req_id = a7738156-be03-48c6-a58c-b1cf055c4c03 response_code = 500 uri = /omicron-zones
13422024-05-15T07:09:25.159ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "a7738156-be03-48c6-a58c-b1cf055c4c03", "content-length": "124", "date": "Wed, 15 May 2024 07:09:24 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "a7738156-be03-48c6-a58c-b1cf055c4c03" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 7.9695987s sled_address = [fd00:1122:3344:101::1]:12345
13432024-05-15T07:09:27.230ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 648.441642ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:426 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
13442024-05-15T07:09:33.129ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 952adea2-114c-4099-9bd9-86f261fe9687, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f2afc369-fa05-4e43-9c24-aaf26564f459, kind: Service { id: 952adea2-114c-4099-9bd9-86f261fe9687 }, name: Name("ntp-952adea2-114c-4099-9bd9-86f261fe9687"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 18])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } } } }, OmicronZoneConfig { id: c7d123b0-1fe2-4336-ad94-79216ae10c14, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9035da45-fa42-447f-985d-89ad334749a3, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } } } }, OmicronZoneConfig { id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
13452024-05-15T07:09:33.130ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13462024-05-15T07:09:33.130ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13472024-05-15T07:09:33.153ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13482024-05-15T07:09:33.153ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
13492024-05-15T07:09:33.174ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13502024-05-15T07:09:33.174ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13512024-05-15T07:09:33.193ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13522024-05-15T07:09:33.193ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13532024-05-15T07:09:33.213ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13542024-05-15T07:09:33.213ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13552024-05-15T07:09:33.233ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13562024-05-15T07:09:33.233ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13572024-05-15T07:09:33.242ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:35017
13582024-05-15T07:09:33.242ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 101 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:35017 req_id = e17b0547-f814-4e8e-8ace-a5e1294af997 response_code = 200 uri = /baseboard
13592024-05-15T07:09:33.252ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13602024-05-15T07:09:33.253ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13612024-05-15T07:09:33.272ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13622024-05-15T07:09:33.272ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13632024-05-15T07:09:33.290ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
13642024-05-15T07:09:33.291ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
13652024-05-15T07:09:33.291ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
13662024-05-15T07:09:33.291ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
13672024-05-15T07:09:33.324ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
13682024-05-15T07:09:33.324ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
13692024-05-15T07:09:33.324ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
13702024-05-15T07:09:33.324ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
13712024-05-15T07:09:33.324ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
13722024-05-15T07:09:33.325ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9", TimeNotSynchronized), ("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14", TimeNotSynchronized), ("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3", TimeNotSynchronized), ("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364", TimeNotSynchronized), ("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 194942 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46219 req_id = f3b51397-f65b-48e2-82fd-3553e1e78d2f response_code = 500 uri = /omicron-zones
13732024-05-15T07:09:33.325ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "f3b51397-f65b-48e2-82fd-3553e1e78d2f", "content-length": "124", "date": "Wed, 15 May 2024 07:09:33 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "f3b51397-f65b-48e2-82fd-3553e1e78d2f" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 6.279875667s sled_address = [fd00:1122:3344:101::1]:12345
13742024-05-15T07:09:39.606ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 952adea2-114c-4099-9bd9-86f261fe9687, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f2afc369-fa05-4e43-9c24-aaf26564f459, kind: Service { id: 952adea2-114c-4099-9bd9-86f261fe9687 }, name: Name("ntp-952adea2-114c-4099-9bd9-86f261fe9687"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 18])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } } } }, OmicronZoneConfig { id: c7d123b0-1fe2-4336-ad94-79216ae10c14, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9035da45-fa42-447f-985d-89ad334749a3, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } } } }, OmicronZoneConfig { id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
13752024-05-15T07:09:39.607ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13762024-05-15T07:09:39.608ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13772024-05-15T07:09:39.630ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13782024-05-15T07:09:39.630ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
13792024-05-15T07:09:39.650ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13802024-05-15T07:09:39.650ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13812024-05-15T07:09:39.669ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13822024-05-15T07:09:39.669ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13832024-05-15T07:09:39.689ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13842024-05-15T07:09:39.689ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13852024-05-15T07:09:39.708ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13862024-05-15T07:09:39.708ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13872024-05-15T07:09:39.727ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13882024-05-15T07:09:39.727ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13892024-05-15T07:09:39.746ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13902024-05-15T07:09:39.746ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13912024-05-15T07:09:39.764ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
13922024-05-15T07:09:39.765ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
13932024-05-15T07:09:39.765ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
13942024-05-15T07:09:39.765ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
13952024-05-15T07:09:39.799ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
13962024-05-15T07:09:39.799ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
13972024-05-15T07:09:39.799ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
13982024-05-15T07:09:39.799ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
13992024-05-15T07:09:39.799ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
14002024-05-15T07:09:39.799ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9", TimeNotSynchronized), ("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3", TimeNotSynchronized), ("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364", TimeNotSynchronized), ("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14", TimeNotSynchronized), ("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 192427 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46219 req_id = bd23c28d-fcea-4ce3-a07e-a3d37f7bb73d response_code = 500 uri = /omicron-zones
14012024-05-15T07:09:39.799ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "bd23c28d-fcea-4ce3-a07e-a3d37f7bb73d", "content-length": "124", "date": "Wed, 15 May 2024 07:09:39 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "bd23c28d-fcea-4ce3-a07e-a3d37f7bb73d" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 12.93951738s sled_address = [fd00:1122:3344:101::1]:12345
14022024-05-15T07:09:42.913ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 1.036520077s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:426 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
14032024-05-15T07:09:52.803ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 952adea2-114c-4099-9bd9-86f261fe9687, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f2afc369-fa05-4e43-9c24-aaf26564f459, kind: Service { id: 952adea2-114c-4099-9bd9-86f261fe9687 }, name: Name("ntp-952adea2-114c-4099-9bd9-86f261fe9687"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 18])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } } } }, OmicronZoneConfig { id: c7d123b0-1fe2-4336-ad94-79216ae10c14, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9035da45-fa42-447f-985d-89ad334749a3, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } } } }, OmicronZoneConfig { id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
14042024-05-15T07:09:52.803ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14052024-05-15T07:09:52.803ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14062024-05-15T07:09:52.826ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14072024-05-15T07:09:52.826ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
14082024-05-15T07:09:52.845ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14092024-05-15T07:09:52.845ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14102024-05-15T07:09:52.864ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14112024-05-15T07:09:52.864ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14122024-05-15T07:09:52.881ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14132024-05-15T07:09:52.882ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14142024-05-15T07:09:52.900ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14152024-05-15T07:09:52.900ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14162024-05-15T07:09:52.918ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14172024-05-15T07:09:52.918ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14182024-05-15T07:09:52.936ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14192024-05-15T07:09:52.936ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14202024-05-15T07:09:52.954ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
14212024-05-15T07:09:52.954ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
14222024-05-15T07:09:52.954ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
14232024-05-15T07:09:52.954ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
14242024-05-15T07:09:52.988ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
14252024-05-15T07:09:52.988ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
14262024-05-15T07:09:52.988ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
14272024-05-15T07:09:52.988ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
14282024-05-15T07:09:52.988ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
14292024-05-15T07:09:52.988ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015", TimeNotSynchronized), ("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14", TimeNotSynchronized), ("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9", TimeNotSynchronized), ("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3", TimeNotSynchronized), ("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 185022 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46219 req_id = 52227679-a170-4104-9267-bb76fbea8e80 response_code = 500 uri = /omicron-zones
14302024-05-15T07:09:52.988ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "52227679-a170-4104-9267-bb76fbea8e80", "content-length": "124", "date": "Wed, 15 May 2024 07:09:52 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "52227679-a170-4104-9267-bb76fbea8e80" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 8.985632361s sled_address = [fd00:1122:3344:101::1]:12345
14312024-05-15T07:09:59.031ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 5.907476941s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:426 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
14322024-05-15T07:10:01.951ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 952adea2-114c-4099-9bd9-86f261fe9687, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f2afc369-fa05-4e43-9c24-aaf26564f459, kind: Service { id: 952adea2-114c-4099-9bd9-86f261fe9687 }, name: Name("ntp-952adea2-114c-4099-9bd9-86f261fe9687"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 18])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } } } }, OmicronZoneConfig { id: c7d123b0-1fe2-4336-ad94-79216ae10c14, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9035da45-fa42-447f-985d-89ad334749a3, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } } } }, OmicronZoneConfig { id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
14332024-05-15T07:10:01.951ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14342024-05-15T07:10:01.951ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14352024-05-15T07:10:01.972ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14362024-05-15T07:10:01.972ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
14372024-05-15T07:10:01.992ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14382024-05-15T07:10:01.992ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14392024-05-15T07:10:02.012ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14402024-05-15T07:10:02.012ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14412024-05-15T07:10:02.030ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14422024-05-15T07:10:02.030ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14432024-05-15T07:10:02.050ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14442024-05-15T07:10:02.050ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14452024-05-15T07:10:02.068ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14462024-05-15T07:10:02.068ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14472024-05-15T07:10:02.087ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14482024-05-15T07:10:02.087ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14492024-05-15T07:10:02.105ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
14502024-05-15T07:10:02.105ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
14512024-05-15T07:10:02.105ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
14522024-05-15T07:10:02.105ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
14532024-05-15T07:10:02.137ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
14542024-05-15T07:10:02.137ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
14552024-05-15T07:10:02.137ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
14562024-05-15T07:10:02.137ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
14572024-05-15T07:10:02.137ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
14582024-05-15T07:10:02.137ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3", TimeNotSynchronized), ("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364", TimeNotSynchronized), ("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14", TimeNotSynchronized), ("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9", TimeNotSynchronized), ("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 186515 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46219 req_id = d1a65eb4-8eab-4f08-9485-288e93121900 response_code = 500 uri = /omicron-zones
14592024-05-15T07:10:02.137ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "d1a65eb4-8eab-4f08-9485-288e93121900", "content-length": "124", "date": "Wed, 15 May 2024 07:10:01 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "d1a65eb4-8eab-4f08-9485-288e93121900" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 14.726985305s sled_address = [fd00:1122:3344:101::1]:12345
14602024-05-15T07:10:03.311ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:48092
14612024-05-15T07:10:03.312ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 62 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:48092 req_id = d532c044-f0b2-4dc7-ada1-548a4203a694 response_code = 200 uri = /baseboard
14622024-05-15T07:10:16.825ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 952adea2-114c-4099-9bd9-86f261fe9687, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f2afc369-fa05-4e43-9c24-aaf26564f459, kind: Service { id: 952adea2-114c-4099-9bd9-86f261fe9687 }, name: Name("ntp-952adea2-114c-4099-9bd9-86f261fe9687"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 18])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } } } }, OmicronZoneConfig { id: c7d123b0-1fe2-4336-ad94-79216ae10c14, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9035da45-fa42-447f-985d-89ad334749a3, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } } } }, OmicronZoneConfig { id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
14632024-05-15T07:10:16.826ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14642024-05-15T07:10:16.826ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14652024-05-15T07:10:16.847ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14662024-05-15T07:10:16.847ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
14672024-05-15T07:10:16.866ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14682024-05-15T07:10:16.866ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14692024-05-15T07:10:16.884ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14702024-05-15T07:10:16.884ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14712024-05-15T07:10:16.903ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14722024-05-15T07:10:16.903ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14732024-05-15T07:10:16.923ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14742024-05-15T07:10:16.923ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14752024-05-15T07:10:16.943ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14762024-05-15T07:10:16.943ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14772024-05-15T07:10:16.962ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14782024-05-15T07:10:16.962ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14792024-05-15T07:10:16.980ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
14802024-05-15T07:10:16.980ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
14812024-05-15T07:10:16.980ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
14822024-05-15T07:10:16.980ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
14832024-05-15T07:10:17.012ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
14842024-05-15T07:10:17.012ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
14852024-05-15T07:10:17.012ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
14862024-05-15T07:10:17.012ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
14872024-05-15T07:10:17.012ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
14882024-05-15T07:10:17.012ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015", TimeNotSynchronized), ("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3", TimeNotSynchronized), ("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364", TimeNotSynchronized), ("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9", TimeNotSynchronized), ("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 187402 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46219 req_id = 222904d9-f1cd-4938-842c-ae8b29a1b6e3 response_code = 500 uri = /omicron-zones
14892024-05-15T07:10:17.013ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "222904d9-f1cd-4938-842c-ae8b29a1b6e3", "content-length": "124", "date": "Wed, 15 May 2024 07:10:17 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "222904d9-f1cd-4938-842c-ae8b29a1b6e3" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 8.884581932s sled_address = [fd00:1122:3344:101::1]:12345
14902024-05-15T07:10:19.920ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 6.250521478s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:426 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
14912024-05-15T07:10:25.874ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 952adea2-114c-4099-9bd9-86f261fe9687, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f2afc369-fa05-4e43-9c24-aaf26564f459, kind: Service { id: 952adea2-114c-4099-9bd9-86f261fe9687 }, name: Name("ntp-952adea2-114c-4099-9bd9-86f261fe9687"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 18])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } } } }, OmicronZoneConfig { id: c7d123b0-1fe2-4336-ad94-79216ae10c14, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9035da45-fa42-447f-985d-89ad334749a3, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } } } }, OmicronZoneConfig { id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
14922024-05-15T07:10:25.874ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14932024-05-15T07:10:25.874ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14942024-05-15T07:10:25.897ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14952024-05-15T07:10:25.897ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
14962024-05-15T07:10:25.918ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14972024-05-15T07:10:25.918ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14982024-05-15T07:10:25.937ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14992024-05-15T07:10:25.937ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15002024-05-15T07:10:25.957ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15012024-05-15T07:10:25.957ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15022024-05-15T07:10:25.975ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15032024-05-15T07:10:25.976ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15042024-05-15T07:10:25.994ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15052024-05-15T07:10:25.994ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15062024-05-15T07:10:26.012ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15072024-05-15T07:10:26.012ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15082024-05-15T07:10:26.031ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
15092024-05-15T07:10:26.031ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
15102024-05-15T07:10:26.031ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
15112024-05-15T07:10:26.031ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
15122024-05-15T07:10:26.064ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
15132024-05-15T07:10:26.064ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
15142024-05-15T07:10:26.064ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
15152024-05-15T07:10:26.064ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
15162024-05-15T07:10:26.064ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
15172024-05-15T07:10:26.064ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9", TimeNotSynchronized), ("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14", TimeNotSynchronized), ("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015", TimeNotSynchronized), ("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3", TimeNotSynchronized), ("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 190529 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46219 req_id = 9e6652a9-7858-47c6-a022-ed7c9ce2ffd3 response_code = 500 uri = /omicron-zones
15182024-05-15T07:10:26.064ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "9e6652a9-7858-47c6-a022-ed7c9ce2ffd3", "content-length": "124", "date": "Wed, 15 May 2024 07:10:25 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "9e6652a9-7858-47c6-a022-ed7c9ce2ffd3" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 11.853351532s sled_address = [fd00:1122:3344:101::1]:12345
15192024-05-15T07:10:33.262ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:45586
15202024-05-15T07:10:33.262ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 46 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:45586 req_id = 7141a425-6bde-4eff-b697-c1954bf104dc response_code = 200 uri = /baseboard
15212024-05-15T07:10:37.886ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 952adea2-114c-4099-9bd9-86f261fe9687, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f2afc369-fa05-4e43-9c24-aaf26564f459, kind: Service { id: 952adea2-114c-4099-9bd9-86f261fe9687 }, name: Name("ntp-952adea2-114c-4099-9bd9-86f261fe9687"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 18])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } } } }, OmicronZoneConfig { id: c7d123b0-1fe2-4336-ad94-79216ae10c14, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9035da45-fa42-447f-985d-89ad334749a3, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } } } }, OmicronZoneConfig { id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
15222024-05-15T07:10:37.887ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15232024-05-15T07:10:37.887ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15242024-05-15T07:10:37.909ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15252024-05-15T07:10:37.909ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
15262024-05-15T07:10:37.928ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15272024-05-15T07:10:37.928ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15282024-05-15T07:10:37.948ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15292024-05-15T07:10:37.948ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15302024-05-15T07:10:37.967ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15312024-05-15T07:10:37.968ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15322024-05-15T07:10:37.987ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15332024-05-15T07:10:37.987ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15342024-05-15T07:10:38.008ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15352024-05-15T07:10:38.008ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15362024-05-15T07:10:38.027ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15372024-05-15T07:10:38.027ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15382024-05-15T07:10:38.046ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
15392024-05-15T07:10:38.046ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
15402024-05-15T07:10:38.046ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
15412024-05-15T07:10:38.046ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
15422024-05-15T07:10:38.079ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
15432024-05-15T07:10:38.079ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
15442024-05-15T07:10:38.079ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
15452024-05-15T07:10:38.079ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
15462024-05-15T07:10:38.079ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
15472024-05-15T07:10:38.079ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9", TimeNotSynchronized), ("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364", TimeNotSynchronized), ("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3", TimeNotSynchronized), ("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14", TimeNotSynchronized), ("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 193536 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46219 req_id = 87e360d0-8300-42ab-9b80-00ad1d96f466 response_code = 500 uri = /omicron-zones
15482024-05-15T07:10:38.080ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "87e360d0-8300-42ab-9b80-00ad1d96f466", "content-length": "124", "date": "Wed, 15 May 2024 07:10:38 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "87e360d0-8300-42ab-9b80-00ad1d96f466" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 13.888696225s sled_address = [fd00:1122:3344:101::1]:12345
15492024-05-15T07:10:41.151ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 23.476752366s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:426 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
15502024-05-15T07:10:51.932ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 952adea2-114c-4099-9bd9-86f261fe9687, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f2afc369-fa05-4e43-9c24-aaf26564f459, kind: Service { id: 952adea2-114c-4099-9bd9-86f261fe9687 }, name: Name("ntp-952adea2-114c-4099-9bd9-86f261fe9687"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 18])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } } } }, OmicronZoneConfig { id: c7d123b0-1fe2-4336-ad94-79216ae10c14, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9035da45-fa42-447f-985d-89ad334749a3, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } } } }, OmicronZoneConfig { id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
15512024-05-15T07:10:51.933ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15522024-05-15T07:10:51.933ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15532024-05-15T07:10:51.954ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15542024-05-15T07:10:51.954ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
15552024-05-15T07:10:51.972ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15562024-05-15T07:10:51.972ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15572024-05-15T07:10:51.990ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15582024-05-15T07:10:51.990ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15592024-05-15T07:10:52.007ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15602024-05-15T07:10:52.007ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15612024-05-15T07:10:52.025ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15622024-05-15T07:10:52.025ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15632024-05-15T07:10:52.042ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15642024-05-15T07:10:52.042ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15652024-05-15T07:10:52.060ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15662024-05-15T07:10:52.060ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15672024-05-15T07:10:52.077ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
15682024-05-15T07:10:52.077ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
15692024-05-15T07:10:52.077ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
15702024-05-15T07:10:52.078ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
15712024-05-15T07:10:52.109ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
15722024-05-15T07:10:52.109ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
15732024-05-15T07:10:52.109ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
15742024-05-15T07:10:52.109ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
15752024-05-15T07:10:52.109ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
15762024-05-15T07:10:52.109ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015", TimeNotSynchronized), ("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14", TimeNotSynchronized), ("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3", TimeNotSynchronized), ("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9", TimeNotSynchronized), ("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 177019 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46219 req_id = 4af6b0ca-c76e-4ca5-9cda-41b92be9948c response_code = 500 uri = /omicron-zones
15772024-05-15T07:10:52.109ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "4af6b0ca-c76e-4ca5-9cda-41b92be9948c", "content-length": "124", "date": "Wed, 15 May 2024 07:10:51 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "4af6b0ca-c76e-4ca5-9cda-41b92be9948c" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 29.427935156s sled_address = [fd00:1122:3344:101::1]:12345
15782024-05-15T07:11:03.216ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:46512
15792024-05-15T07:11:03.216ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 73 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:46512 req_id = a1ede15f-1eae-4837-b47b-ca4642768282 response_code = 200 uri = /baseboard
15802024-05-15T07:11:19.559ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 29.914242547s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:426 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
15812024-05-15T07:11:21.459ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 952adea2-114c-4099-9bd9-86f261fe9687, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f2afc369-fa05-4e43-9c24-aaf26564f459, kind: Service { id: 952adea2-114c-4099-9bd9-86f261fe9687 }, name: Name("ntp-952adea2-114c-4099-9bd9-86f261fe9687"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 18])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } } } }, OmicronZoneConfig { id: c7d123b0-1fe2-4336-ad94-79216ae10c14, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9035da45-fa42-447f-985d-89ad334749a3, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } } } }, OmicronZoneConfig { id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
15822024-05-15T07:11:21.459ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15832024-05-15T07:11:21.460ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15842024-05-15T07:11:21.482ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15852024-05-15T07:11:21.482ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
15862024-05-15T07:11:21.503ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15872024-05-15T07:11:21.503ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15882024-05-15T07:11:21.522ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15892024-05-15T07:11:21.522ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15902024-05-15T07:11:21.541ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15912024-05-15T07:11:21.541ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15922024-05-15T07:11:21.560ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15932024-05-15T07:11:21.560ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15942024-05-15T07:11:21.579ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15952024-05-15T07:11:21.579ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15962024-05-15T07:11:21.597ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15972024-05-15T07:11:21.597ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15982024-05-15T07:11:21.614ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
15992024-05-15T07:11:21.614ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
16002024-05-15T07:11:21.615ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
16012024-05-15T07:11:21.615ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
16022024-05-15T07:11:21.697ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
16032024-05-15T07:11:21.698ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
16042024-05-15T07:11:21.698ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
16052024-05-15T07:11:21.698ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
16062024-05-15T07:11:21.698ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
16072024-05-15T07:11:21.788ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015 file = illumos-utils/src/zone.rs:315
16082024-05-15T07:11:21.793ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9 file = illumos-utils/src/zone.rs:315
16092024-05-15T07:11:21.797ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14 file = illumos-utils/src/zone.rs:315
16102024-05-15T07:11:21.801ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3 file = illumos-utils/src/zone.rs:315
16112024-05-15T07:11:21.806ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364 file = illumos-utils/src/zone.rs:315
16122024-05-15T07:11:21.810ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015 file = illumos-utils/src/zone.rs:354
16132024-05-15T07:11:21.815ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9 file = illumos-utils/src/zone.rs:354
16142024-05-15T07:11:21.819ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14 file = illumos-utils/src/zone.rs:354
16152024-05-15T07:11:21.824ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3 file = illumos-utils/src/zone.rs:354
16162024-05-15T07:11:21.828ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364 file = illumos-utils/src/zone.rs:354
16172024-05-15T07:11:24.586ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService9"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::7"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::7"/> <propval type="astring" name="listen_port" value="32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
16182024-05-15T07:11:24.586ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
16192024-05-15T07:11:24.617ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService6"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::4"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::4"/> <propval type="astring" name="listen_port" value="32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
16202024-05-15T07:11:24.618ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
16212024-05-15T07:11:24.670ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService7"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::5"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::5"/> <propval type="astring" name="listen_port" value="32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
16222024-05-15T07:11:24.670ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
16232024-05-15T07:11:24.700ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService8"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::6"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::6"/> <propval type="astring" name="listen_port" value="32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
16242024-05-15T07:11:24.700ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
16252024-05-15T07:11:24.707ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService5"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::3"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::3"/> <propval type="astring" name="listen_port" value="32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
16262024-05-15T07:11:24.707ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
16272024-05-15T07:11:24.739ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364") failed: Property not found. retry in 25.957277ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
16282024-05-15T07:11:24.760ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9") failed: Property not found. retry in 30.158492ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
16292024-05-15T07:11:24.774ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364") failed: Property not found. retry in 98.91011ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
16302024-05-15T07:11:24.799ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9") failed: Property not found. retry in 110.053605ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
16312024-05-15T07:11:24.830ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14") failed: Property not found. retry in 26.200463ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
16322024-05-15T07:11:24.874ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14") failed: Property not found. retry in 118.247142ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
16332024-05-15T07:11:24.890ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364") failed: Property not found. retry in 208.97809ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
16342024-05-15T07:11:25.002ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9") failed: Property not found. retry in 126.928496ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
16352024-05-15T07:11:25.949ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015") failed: Property not found. retry in 55.062582ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
16362024-05-15T07:11:26.008ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14") failed: Property not found. retry in 255.533843ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
16372024-05-15T07:11:26.656ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3") failed: Property not found. retry in 62.558119ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
16382024-05-15T07:11:27.235ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364") failed: Property not found. retry in 341.651539ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
16392024-05-15T07:11:27.450ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015") failed: Property not found. retry in 140.64136ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
16402024-05-15T07:11:27.501ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9") failed: Property not found. retry in 316.493892ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
16412024-05-15T07:11:27.518ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14") failed: Property not found. retry in 538.016106ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
16422024-05-15T07:11:27.534ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3") failed: Property not found. retry in 103.732415ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
16432024-05-15T07:11:27.589ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364") failed: Property not found. retry in 658.982117ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
16442024-05-15T07:11:27.602ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015") failed: Property not found. retry in 158.318981ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
16452024-05-15T07:11:27.649ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3") failed: Property not found. retry in 218.258222ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
16462024-05-15T07:11:27.772ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015") failed: Property not found. retry in 310.254095ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
16472024-05-15T07:11:27.828ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9") failed: Property not found. retry in 563.716868ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
16482024-05-15T07:11:27.877ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3") failed: Property not found. retry in 314.45902ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
16492024-05-15T07:11:28.069ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14") failed: Property not found. retry in 749.94118ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
16502024-05-15T07:11:28.093ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015") failed: Property not found. retry in 822.716309ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
16512024-05-15T07:11:28.202ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3") failed: Property not found. retry in 433.775582ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
16522024-05-15T07:11:28.255ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364") failed: Property not found. retry in 1.259350939s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
16532024-05-15T07:11:28.399ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9") failed: Property not found. retry in 1.472003621s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
16542024-05-15T07:11:28.644ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3") failed: Property not found. retry in 672.026583ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
16552024-05-15T07:11:28.826ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14") failed: Property not found. retry in 1.370054446s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
16562024-05-15T07:11:28.923ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015") failed: Property not found. retry in 841.249836ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
16572024-05-15T07:11:29.323ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3") failed: Property not found. retry in 937.460185ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
16582024-05-15T07:11:29.522ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364") failed: Property not found. retry in 1.35119037s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
16592024-05-15T07:11:29.771ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015") failed: Property not found. retry in 557.166031ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
16602024-05-15T07:11:29.875ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9") failed: Property not found. retry in 1.337678125s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
16612024-05-15T07:11:30.203ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14") failed: Property not found. retry in 1.368348792s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
16622024-05-15T07:11:30.267ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3") failed: Property not found. retry in 1.042648909s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
16632024-05-15T07:11:30.335ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015") failed: Property not found. retry in 1.205500267s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
16642024-05-15T07:11:30.878ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364") failed: Property not found. retry in 952.148133ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
16652024-05-15T07:11:31.218ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9") failed: Property not found. retry in 1.159055229s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
16662024-05-15T07:11:31.315ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3") failed: Property not found. retry in 1.435099279s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
16672024-05-15T07:11:31.547ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015") failed: Property not found. retry in 676.030687ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
16682024-05-15T07:11:31.576ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14") failed: Property not found. retry in 1.215315155s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
16692024-05-15T07:11:31.837ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364") failed: Property not found. retry in 1.265614948s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
16702024-05-15T07:11:32.231ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015") failed: Property not found. retry in 584.997008ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
16712024-05-15T07:11:32.384ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9") failed: Property not found. retry in 1.253841954s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
16722024-05-15T07:11:32.756ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3") failed: Property not found. retry in 1.148210669s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
16732024-05-15T07:11:32.798ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14") failed: Property not found. retry in 578.659749ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
16742024-05-15T07:11:32.825ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015") failed: Property not found. retry in 500.754411ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
16752024-05-15T07:11:33.109ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364") failed: Property not found. retry in 1.189335243s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
16762024-05-15T07:11:33.169ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:55222
16772024-05-15T07:11:33.169ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 73 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:55222 req_id = 811fb532-33e0-40c4-a521-fef1ba63c7b2 response_code = 200 uri = /baseboard
16782024-05-15T07:11:33.335ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015") failed: Property not found. retry in 1.417304303s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
16792024-05-15T07:11:33.386ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14") failed: Property not found. retry in 653.437501ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
16802024-05-15T07:11:33.648ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9") failed: Property not found. retry in 562.716695ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
16812024-05-15T07:11:33.913ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3") failed: Property not found. retry in 919.566228ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
16822024-05-15T07:11:34.048ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14") failed: Property not found. retry in 1.361696063s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
16832024-05-15T07:11:34.218ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9") failed: Property not found. retry in 671.668595ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
16842024-05-15T07:11:34.302ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364") failed: Property not found. retry in 1.03296041s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
16852024-05-15T07:11:34.758ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015") failed: Property not found. retry in 1.325834561s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
16862024-05-15T07:11:34.838ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3") failed: Property not found. retry in 1.399986764s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
16872024-05-15T07:11:34.896ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9") failed: Property not found. retry in 662.064363ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
16882024-05-15T07:11:35.341ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364") failed: Property not found. retry in 1.292759464s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
16892024-05-15T07:11:35.417ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14") failed: Property not found. retry in 1.436275437s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
16902024-05-15T07:11:35.568ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9") failed: Property not found. retry in 1.449790263s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
16912024-05-15T07:11:36.092ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015") failed: Property not found. retry in 1.366299925s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
16922024-05-15T07:11:36.244ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3") failed: Property not found. retry in 1.156617879s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
16932024-05-15T07:11:36.640ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364") failed: Property not found. retry in 1.063676424s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
16942024-05-15T07:11:36.861ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14") failed: Property not found. retry in 1.413317327s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
16952024-05-15T07:11:37.024ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9") failed: Property not found. retry in 1.030118813s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
16962024-05-15T07:11:38.062ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9") failed: Property not found. retry in 856.609269ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
16972024-05-15T07:11:38.928ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9") failed: Property not found. retry in 1.079181332s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
16982024-05-15T07:11:40.017ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9") failed: Property not found. retry in 1.216604685s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
16992024-05-15T07:11:41.915ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9") failed: Property not found. retry in 894.910559ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
17002024-05-15T07:11:42.820ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9") failed: Property not found. retry in 754.315926ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
17012024-05-15T07:11:43.586ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9") failed: Property not found. retry in 715.746732ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
17022024-05-15T07:11:44.311ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9") failed: Property not found. retry in 1.099954363s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
17032024-05-15T07:11:45.421ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9") failed: Property not found. retry in 786.999032ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
17042024-05-15T07:11:46.218ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9") failed: Property not found. retry in 591.426581ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
17052024-05-15T07:11:46.819ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9") failed: Property not found. retry in 756.991228ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
17062024-05-15T07:11:47.597ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_cockroachdb_2ec6ec59-0609-4ca3-a298-dfc71ccab015
17072024-05-15T07:11:47.597ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_cockroachdb_9b6e3ef6-b14c-455b-8400-1ee63ad5adb9
17082024-05-15T07:11:47.597ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_cockroachdb_c7d123b0-1fe2-4336-ad94-79216ae10c14
17092024-05-15T07:11:47.597ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_cockroachdb_9035da45-fa42-447f-985d-89ad334749a3
17102024-05-15T07:11:47.597ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_cockroachdb_76c39d73-e744-4aa3-9c13-dc1e7e8eb364
17112024-05-15T07:11:47.597ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
17122024-05-15T07:11:47.598ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
17132024-05-15T07:11:47.707ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 26295351 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46219 req_id = 985021d9-8241-43da-9ea6-6213bf73e147 response_code = 204 uri = /omicron-zones
17142024-05-15T07:11:47.739ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:54422
17152024-05-15T07:11:47.767ZINFOSledAgent (ServiceManager): Initializing CRDB Cluster - sending request to [fd00:1122:3344:101::3]:32221 file = sled-agent/src/services.rs:3437
17162024-05-15T07:11:54.294ZINFOSledAgent (ServiceManager): Formatting CRDB file = sled-agent/src/services.rs:3455
17172024-05-15T07:11:58.073ZINFOSledAgent (ServiceManager): Formatting CRDB - Completed file = sled-agent/src/services.rs:3478
17182024-05-15T07:11:58.073ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 10333757 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:54422 req_id = 0927ca2a-e7e8-4b09-ad02-9dd0b8f66b7d response_code = 204 uri = /cockroachdb
17192024-05-15T07:11:58.135ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: adb60b2d-67ac-445a-817b-3e6611ca7982, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 952adea2-114c-4099-9bd9-86f261fe9687, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: f2afc369-fa05-4e43-9c24-aaf26564f459, kind: Service { id: 952adea2-114c-4099-9bd9-86f261fe9687 }, name: Name("ntp-952adea2-114c-4099-9bd9-86f261fe9687"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 18])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } } } }, OmicronZoneConfig { id: c7d123b0-1fe2-4336-ad94-79216ae10c14, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9035da45-fa42-447f-985d-89ad334749a3, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } } } }, OmicronZoneConfig { id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 12908044-fccf-4c46-8e7c-c2a0be1bfb25, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: fe506ce4-c84e-4ef5-9ed5-5c96a847dd43, kind: Service { id: 12908044-fccf-4c46-8e7c-c2a0be1bfb25 }, name: Name("external-dns-12908044-fccf-4c46-8e7c-c2a0be1bfb25"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 70])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: e5538333-fcad-480c-9181-aa7e94aaf834, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 2e30c7c4-8de6-4cbd-8a3d-052bb77d3985, kind: Service { id: e5538333-fcad-480c-9181-aa7e94aaf834 }, name: Name("external-dns-e5538333-fcad-480c-9181-aa7e94aaf834"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 204, 142])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 99861af6-438d-4bef-ba93-632645b46d2f, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 38eee15b-cb30-4e2c-b5c3-f5511aac6743, kind: Service { id: 99861af6-438d-4bef-ba93-632645b46d2f }, name: Name("nexus-99861af6-438d-4bef-ba93-632645b46d2f"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 222, 14])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 9cebcc66-fa56-4e34-af63-2c7ac52ff959, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 9b63bcc5-cf23-47dc-87dc-38f5ac4bcfd6, kind: Service { id: 9cebcc66-fa56-4e34-af63-2c7ac52ff959 }, name: Name("nexus-9cebcc66-fa56-4e34-af63-2c7ac52ff959"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 234, 140])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 88dead63-2254-4036-85c5-6efbd3602a0d, kind: Service { id: feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3 }, name: Name("nexus-feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 207, 1])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 7d8b5024-88b0-4173-8867-4e833f05550e, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 7ce930b4-1bea-41ce-8c9a-2aa16fd16d05, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 638e5dab-76c4-4b78-b15a-a7faf399d1d8, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 42687f02-f9e0-4e2e-8b1e-8fcc2648d21d, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: d2e82ff1-e608-4c9b-8b24-06c6a00e8c2a, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: ae891598-60c4-47ca-bd0b-302a9e60a0bc, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7b766ca5-5d0c-459d-bc67-f339bcac986b, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External } } } }, OmicronZoneConfig { id: 03e9085d-76ce-4798-9d35-12ceecd4c607, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External } } } }, OmicronZoneConfig { id: e0bae9f0-075a-41e1-a5e1-22133ae05550, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External } } } }, OmicronZoneConfig { id: d62b80e0-a2d7-4b05-8ee7-7f6a9516881e, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External } } } }, OmicronZoneConfig { id: d41b4327-a68c-4bde-a48b-cfcb75b4c054, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5da68ecb-08f6-4895-925f-903c81000441 (zpool), kind: External } } } }, OmicronZoneConfig { id: ecac0828-7786-4503-af9f-835e69870288, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3b36f3f1-29d0-4d11-8b6b-ae2a89620d43 (zpool), kind: External } } } }, OmicronZoneConfig { id: eece7197-e5eb-4531-85f7-6700f07faf6b, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2780ea55-1799-4c3e-916a-9024b9586eaf (zpool), kind: External } } } }, OmicronZoneConfig { id: c06cd1db-bb07-41fa-af89-0fd52ff0a0a8, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bae019e4-4e95-466e-aa8a-f660660e26c1 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
17202024-05-15T07:11:58.135ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:60075
17212024-05-15T07:11:58.136ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17222024-05-15T07:11:58.136ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 18c3fbe6-7d91-49c3-bac0-64bc0ebaaa62, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: InternalDns }, responder: ... }
17232024-05-15T07:11:58.160ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17242024-05-15T07:11:58.160ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: adb60b2d-67ac-445a-817b-3e6611ca7982, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: InternalDns }, responder: ... }
17252024-05-15T07:11:58.182ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17262024-05-15T07:11:58.182ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 08569023-bf34-46c3-a27b-d9b75bdbd67c, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: InternalDns }, responder: ... }
17272024-05-15T07:11:58.202ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17282024-05-15T07:11:58.203ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 2ec6ec59-0609-4ca3-a298-dfc71ccab015, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17292024-05-15T07:11:58.223ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17302024-05-15T07:11:58.223ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b6e3ef6-b14c-455b-8400-1ee63ad5adb9, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17312024-05-15T07:11:58.244ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17322024-05-15T07:11:58.244ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c7d123b0-1fe2-4336-ad94-79216ae10c14, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17332024-05-15T07:11:58.264ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17342024-05-15T07:11:58.264ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9035da45-fa42-447f-985d-89ad334749a3, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17352024-05-15T07:11:58.284ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17362024-05-15T07:11:58.284ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 76c39d73-e744-4aa3-9c13-dc1e7e8eb364, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17372024-05-15T07:11:58.304ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12908044-fccf-4c46-8e7c-c2a0be1bfb25, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17382024-05-15T07:11:58.304ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12908044-fccf-4c46-8e7c-c2a0be1bfb25, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
17392024-05-15T07:11:58.378ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e5538333-fcad-480c-9181-aa7e94aaf834, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17402024-05-15T07:11:58.378ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: e5538333-fcad-480c-9181-aa7e94aaf834, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: ExternalDns }, responder: ... }
17412024-05-15T07:11:58.454ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7ce930b4-1bea-41ce-8c9a-2aa16fd16d05, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
17422024-05-15T07:11:58.454ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 7ce930b4-1bea-41ce-8c9a-2aa16fd16d05, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
17432024-05-15T07:11:58.527ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ae891598-60c4-47ca-bd0b-302a9e60a0bc, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17442024-05-15T07:11:58.527ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ae891598-60c4-47ca-bd0b-302a9e60a0bc, dataset_name: DatasetName { pool_name: ZpoolName { id: 4e6ef30f-a7d9-4acc-99aa-59ae1084fa14 (zpool), kind: External }, kind: Crucible }, responder: ... }
17452024-05-15T07:11:58.598ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7b766ca5-5d0c-459d-bc67-f339bcac986b, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17462024-05-15T07:11:58.598ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 7b766ca5-5d0c-459d-bc67-f339bcac986b, dataset_name: DatasetName { pool_name: ZpoolName { id: 12d428b9-eedd-40c0-8928-1d92a73f181d (zpool), kind: External }, kind: Crucible }, responder: ... }
17472024-05-15T07:11:58.667ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 03e9085d-76ce-4798-9d35-12ceecd4c607, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17482024-05-15T07:11:58.667ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 03e9085d-76ce-4798-9d35-12ceecd4c607, dataset_name: DatasetName { pool_name: ZpoolName { id: 08d4507c-e72e-4afd-9f03-5e28f4ad0909 (zpool), kind: External }, kind: Crucible }, responder: ... }
17492024-05-15T07:11:58.745ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e0bae9f0-075a-41e1-a5e1-22133ae05550, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17502024-05-15T07:11:58.746ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: e0bae9f0-075a-41e1-a5e1-22133ae05550, dataset_name: DatasetName { pool_name: ZpoolName { id: 7ba4af02-bf8f-47c6-af79-d69fdad5d4bf (zpool), kind: External }, kind: Crucible }, responder: ... }
17512024-05-15T07:11:58.819ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d62b80e0-a2d7-4b05-8ee7-7f6a9516881e, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17522024-05-15T07:11:58.819ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d62b80e0-a2d7-4b05-8ee7-7f6a9516881e, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a187a2-0706-420e-a881-83ff781ad6c1 (zpool), kind: External }, kind: Crucible }, responder: ... }
17532024-05-15T07:11:58.894ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d41b4327-a68c-4bde-a48b-cfcb75b4c054, dataset_name: DatasetName { pool_name: ZpoolName { id: 5da68ecb-08f6-4895-925f-903c81000441 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17542024-05-15T07:11:58.894ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d41b4327-a68c-4bde-a48b-cfcb75b4c054, dataset_name: DatasetName { pool_name: ZpoolName { id: 5da68ecb-08f6-4895-925f-903c81000441 (zpool), kind: External }, kind: Crucible }, responder: ... }
17552024-05-15T07:11:58.961ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ecac0828-7786-4503-af9f-835e69870288, dataset_name: DatasetName { pool_name: ZpoolName { id: 3b36f3f1-29d0-4d11-8b6b-ae2a89620d43 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17562024-05-15T07:11:58.961ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ecac0828-7786-4503-af9f-835e69870288, dataset_name: DatasetName { pool_name: ZpoolName { id: 3b36f3f1-29d0-4d11-8b6b-ae2a89620d43 (zpool), kind: External }, kind: Crucible }, responder: ... }
17572024-05-15T07:11:59.030ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eece7197-e5eb-4531-85f7-6700f07faf6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 2780ea55-1799-4c3e-916a-9024b9586eaf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17582024-05-15T07:11:59.030ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: eece7197-e5eb-4531-85f7-6700f07faf6b, dataset_name: DatasetName { pool_name: ZpoolName { id: 2780ea55-1799-4c3e-916a-9024b9586eaf (zpool), kind: External }, kind: Crucible }, responder: ... }
17592024-05-15T07:11:59.097ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c06cd1db-bb07-41fa-af89-0fd52ff0a0a8, dataset_name: DatasetName { pool_name: ZpoolName { id: bae019e4-4e95-466e-aa8a-f660660e26c1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17602024-05-15T07:11:59.097ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c06cd1db-bb07-41fa-af89-0fd52ff0a0a8, dataset_name: DatasetName { pool_name: ZpoolName { id: bae019e4-4e95-466e-aa8a-f660660e26c1 (zpool), kind: External }, kind: Crucible }, responder: ... }
17612024-05-15T07:11:59.166ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
17622024-05-15T07:11:59.167ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
17632024-05-15T07:11:59.167ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
17642024-05-15T07:11:59.167ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
17652024-05-15T07:11:59.286ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:259
17662024-05-15T07:11:59.299ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:259
17672024-05-15T07:11:59.341ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:259
17682024-05-15T07:11:59.341ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
17692024-05-15T07:11:59.341ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
17702024-05-15T07:11:59.341ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
17712024-05-15T07:11:59.341ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
17722024-05-15T07:11:59.341ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
17732024-05-15T07:11:59.342ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:259
17742024-05-15T07:11:59.357ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
17752024-05-15T07:11:59.357ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
17762024-05-15T07:11:59.591ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:259
17772024-05-15T07:11:59.591ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
17782024-05-15T07:11:59.591ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
17792024-05-15T07:11:59.591ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
17802024-05-15T07:11:59.591ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
17812024-05-15T07:11:59.591ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
17822024-05-15T07:11:59.591ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
17832024-05-15T07:11:59.591ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
17842024-05-15T07:11:59.592ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e file = illumos-utils/src/zone.rs:315
17852024-05-15T07:11:59.597ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8 file = illumos-utils/src/zone.rs:315
17862024-05-15T07:11:59.602ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_7b766ca5-5d0c-459d-bc67-f339bcac986b file = illumos-utils/src/zone.rs:315
17872024-05-15T07:11:59.607ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_03e9085d-76ce-4798-9d35-12ceecd4c607 file = illumos-utils/src/zone.rs:315
17882024-05-15T07:11:59.612ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e file = illumos-utils/src/zone.rs:315
17892024-05-15T07:11:59.618ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_ae891598-60c4-47ca-bd0b-302a9e60a0bc file = illumos-utils/src/zone.rs:315
17902024-05-15T07:11:59.623ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_42687f02-f9e0-4e2e-8b1e-8fcc2648d21d file = illumos-utils/src/zone.rs:315
17912024-05-15T07:11:59.774ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e file = illumos-utils/src/zone.rs:354
17922024-05-15T07:11:59.779ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8 file = illumos-utils/src/zone.rs:354
17932024-05-15T07:11:59.784ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_7b766ca5-5d0c-459d-bc67-f339bcac986b file = illumos-utils/src/zone.rs:354
17942024-05-15T07:11:59.788ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_03e9085d-76ce-4798-9d35-12ceecd4c607 file = illumos-utils/src/zone.rs:354
17952024-05-15T07:11:59.793ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e file = illumos-utils/src/zone.rs:354
17962024-05-15T07:11:59.798ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_ae891598-60c4-47ca-bd0b-302a9e60a0bc file = illumos-utils/src/zone.rs:354
17972024-05-15T07:11:59.803ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_42687f02-f9e0-4e2e-8b1e-8fcc2648d21d file = illumos-utils/src/zone.rs:354
17982024-05-15T07:11:59.808ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054 file = illumos-utils/src/zone.rs:315
17992024-05-15T07:11:59.814ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b file = illumos-utils/src/zone.rs:315
18002024-05-15T07:11:59.819ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8 file = illumos-utils/src/zone.rs:315
18012024-05-15T07:11:59.824ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_d2e82ff1-e608-4c9b-8b24-06c6a00e8c2a file = illumos-utils/src/zone.rs:315
18022024-05-15T07:11:59.831ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_ecac0828-7786-4503-af9f-835e69870288 file = illumos-utils/src/zone.rs:315
18032024-05-15T07:11:59.837ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_e0bae9f0-075a-41e1-a5e1-22133ae05550 file = illumos-utils/src/zone.rs:315
18042024-05-15T07:11:59.844ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_clickhouse_7ce930b4-1bea-41ce-8c9a-2aa16fd16d05 file = illumos-utils/src/zone.rs:315
18052024-05-15T07:11:59.851ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054 file = illumos-utils/src/zone.rs:354
18062024-05-15T07:11:59.856ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b file = illumos-utils/src/zone.rs:354
18072024-05-15T07:11:59.862ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8 file = illumos-utils/src/zone.rs:354
18082024-05-15T07:11:59.868ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_d2e82ff1-e608-4c9b-8b24-06c6a00e8c2a file = illumos-utils/src/zone.rs:354
18092024-05-15T07:11:59.874ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_ecac0828-7786-4503-af9f-835e69870288 file = illumos-utils/src/zone.rs:354
18102024-05-15T07:11:59.880ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_e0bae9f0-075a-41e1-a5e1-22133ae05550 file = illumos-utils/src/zone.rs:354
18112024-05-15T07:11:59.887ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_clickhouse_7ce930b4-1bea-41ce-8c9a-2aa16fd16d05 file = illumos-utils/src/zone.rs:354
18122024-05-15T07:11:59.946ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:276
18132024-05-15T07:12:00.050ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:276
18142024-05-15T07:12:00.163ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:276
18152024-05-15T07:12:00.220ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:276
18162024-05-15T07:12:00.225ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:301
18172024-05-15T07:12:00.225ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:193
18182024-05-15T07:12:00.353ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:394 port = Port { inner: PortInner { name: "opte1", ip: 172.30.2.5, mac: MacAddr6([168, 64, 37, 255, 222, 14]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 }, vnic: "vopte1" } }
18192024-05-15T07:12:00.353ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1290 zone_type = nexus
18202024-05-15T07:12:00.356ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:301
18212024-05-15T07:12:00.356ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:193
18222024-05-15T07:12:00.464ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:394 port = Port { inner: PortInner { name: "opte2", ip: 172.30.2.6, mac: MacAddr6([168, 64, 37, 255, 234, 140]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 }, vnic: "vopte2" } }
18232024-05-15T07:12:00.464ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1290 zone_type = nexus
18242024-05-15T07:12:00.520ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:276
18252024-05-15T07:12:00.522ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:301
18262024-05-15T07:12:00.522ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:193
18272024-05-15T07:12:00.629ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:394 port = Port { inner: PortInner { name: "opte3", ip: 172.30.1.6, mac: MacAddr6([168, 64, 37, 255, 204, 142]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 }, vnic: "vopte3" } }
18282024-05-15T07:12:00.629ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1290 zone_type = external_dns
18292024-05-15T07:12:00.634ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:301
18302024-05-15T07:12:00.634ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:193
18312024-05-15T07:12:00.769ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:394 port = Port { inner: PortInner { name: "opte4", ip: 172.30.1.5, mac: MacAddr6([168, 64, 37, 255, 254, 70]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 }, vnic: "vopte4" } }
18322024-05-15T07:12:00.769ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1290 zone_type = external_dns
18332024-05-15T07:12:00.771ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.2.166, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
18342024-05-15T07:12:00.771ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.2.166, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
18352024-05-15T07:12:00.774ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.2.167, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
18362024-05-15T07:12:00.774ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.2.167, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
18372024-05-15T07:12:00.777ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:301
18382024-05-15T07:12:00.777ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:193
18392024-05-15T07:12:00.865ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:394 port = Port { inner: PortInner { name: "opte5", ip: 172.30.2.7, mac: MacAddr6([168, 64, 37, 255, 207, 1]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 }, vnic: "vopte5" } }
18402024-05-15T07:12:00.865ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1290 zone_type = nexus
18412024-05-15T07:12:00.866ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.2.165, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
18422024-05-15T07:12:00.866ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.2.165, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
18432024-05-15T07:12:00.866ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.2.164, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
18442024-05-15T07:12:00.866ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.2.164, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
18452024-05-15T07:12:00.866ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.2.166, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
18462024-05-15T07:12:00.869ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.2.167, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
18472024-05-15T07:12:00.872ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18482024-05-15T07:12:00.872ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18492024-05-15T07:12:00.872ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.2.165, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
18502024-05-15T07:12:00.874ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.2.164, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
18512024-05-15T07:12:00.877ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18522024-05-15T07:12:00.877ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18532024-05-15T07:12:01.042ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_99861af6-438d-4bef-ba93-632645b46d2f file = illumos-utils/src/zone.rs:315
18542024-05-15T07:12:01.051ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_9cebcc66-fa56-4e34-af63-2c7ac52ff959 file = illumos-utils/src/zone.rs:315
18552024-05-15T07:12:01.061ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.2.168, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
18562024-05-15T07:12:01.061ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.2.168, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
18572024-05-15T07:12:01.061ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_e5538333-fcad-480c-9181-aa7e94aaf834 file = illumos-utils/src/zone.rs:315
18582024-05-15T07:12:01.071ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25 file = illumos-utils/src/zone.rs:315
18592024-05-15T07:12:01.080ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.2.168, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
18602024-05-15T07:12:01.083ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18612024-05-15T07:12:01.127ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_e5538333-fcad-480c-9181-aa7e94aaf834 file = illumos-utils/src/zone.rs:354
18622024-05-15T07:12:01.136ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25 file = illumos-utils/src/zone.rs:354
18632024-05-15T07:12:01.146ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_99861af6-438d-4bef-ba93-632645b46d2f file = illumos-utils/src/zone.rs:354
18642024-05-15T07:12:01.155ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_9cebcc66-fa56-4e34-af63-2c7ac52ff959 file = illumos-utils/src/zone.rs:354
18652024-05-15T07:12:01.164ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3 file = illumos-utils/src/zone.rs:315
18662024-05-15T07:12:01.218ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3 file = illumos-utils/src/zone.rs:354
18672024-05-15T07:12:03.456ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:42916
18682024-05-15T07:12:03.461ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 5974 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:42916 req_id = ac69487d-a473-4478-b014-8079d6ea2a84 response_code = 200 uri = /baseboard
18692024-05-15T07:12:05.216ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_42687f02-f9e0-4e2e-8b1e-8fcc2648d21d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService16"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::10"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::10"/> <propval type="astring" name="listen_port" value="17000"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18702024-05-15T07:12:05.232ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_pantry_42687f02-f9e0-4e2e-8b1e-8fcc2648d21d
18712024-05-15T07:12:05.243ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_03e9085d-76ce-4798-9d35-12ceecd4c607: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService13"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::14"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_08d4507c-e72e-4afd-9f03-5e28f4ad0909/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::14"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="3a0d11c9-b2a2-4a79-876b-36faa9fb4d9f"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18722024-05-15T07:12:05.246ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_03e9085d-76ce-4798-9d35-12ceecd4c607
18732024-05-15T07:12:05.265ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_7b766ca5-5d0c-459d-bc67-f339bcac986b: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService12"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::13"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_12d428b9-eedd-40c0-8928-1d92a73f181d/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::13"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="17ecf68f-f3fc-4c24-b9d0-9bbad0bb0f5f"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18742024-05-15T07:12:05.266ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_7b766ca5-5d0c-459d-bc67-f339bcac986b
18752024-05-15T07:12:05.312ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService11"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::f"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::f"/> <propval type="astring" name="listen_port" value="17000"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18762024-05-15T07:12:05.313ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8
18772024-05-15T07:12:05.574ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_d2e82ff1-e608-4c9b-8b24-06c6a00e8c2a: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService20"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::11"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::11"/> <propval type="astring" name="listen_port" value="17000"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18782024-05-15T07:12:05.574ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_pantry_d2e82ff1-e608-4c9b-8b24-06c6a00e8c2a
18792024-05-15T07:12:05.716ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService19"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::1a"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_bae019e4-4e95-466e-aa8a-f660660e26c1/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::1a"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="8e2c2ebf-8081-4938-81e8-8c81b0b67a2f"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18802024-05-15T07:12:05.716ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8
18812024-05-15T07:12:05.736ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService17"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::17"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_5da68ecb-08f6-4895-925f-903c81000441/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::17"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="979acc2a-4937-4b2a-9089-a4c17553a937"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18822024-05-15T07:12:05.738ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054
18832024-05-15T07:12:05.758ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_e0bae9f0-075a-41e1-a5e1-22133ae05550: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService22"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::15"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_7ba4af02-bf8f-47c6-af79-d69fdad5d4bf/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::15"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="089c3f0c-afa2-449f-8ea8-924037b9db1f"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18842024-05-15T07:12:05.760ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_e0bae9f0-075a-41e1-a5e1-22133ae05550
18852024-05-15T07:12:05.846ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_03e9085d-76ce-4798-9d35-12ceecd4c607") failed: Property not found. retry in 70.913775ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_03e9085d-76ce-4798-9d35-12ceecd4c607
18862024-05-15T07:12:05.881ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8") failed: Property not found. retry in 58.201696ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8
18872024-05-15T07:12:05.909ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7b766ca5-5d0c-459d-bc67-f339bcac986b") failed: Property not found. retry in 65.088482ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7b766ca5-5d0c-459d-bc67-f339bcac986b
18882024-05-15T07:12:05.942ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_42687f02-f9e0-4e2e-8b1e-8fcc2648d21d") failed: Property not found. retry in 35.321707ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_42687f02-f9e0-4e2e-8b1e-8fcc2648d21d
18892024-05-15T07:12:05.945ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService18"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::19"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_2780ea55-1799-4c3e-916a-9024b9586eaf/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::19"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="36afc74d-4c2b-4c9a-aec0-794dbba4f078"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18902024-05-15T07:12:05.971ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8") failed: Property not found. retry in 109.563332ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8
18912024-05-15T07:12:06.093ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_03e9085d-76ce-4798-9d35-12ceecd4c607") failed: Property not found. retry in 98.091036ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_03e9085d-76ce-4798-9d35-12ceecd4c607
18922024-05-15T07:12:06.094ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService14"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::16"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_62a187a2-0706-420e-a881-83ff781ad6c1/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::16"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="5b5fcf9b-c81b-46f9-b73e-97c320d093b8"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18932024-05-15T07:12:06.134ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_42687f02-f9e0-4e2e-8b1e-8fcc2648d21d") failed: Property not found. retry in 57.054824ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_42687f02-f9e0-4e2e-8b1e-8fcc2648d21d
18942024-05-15T07:12:06.134ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b
18952024-05-15T07:12:06.200ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_ecac0828-7786-4503-af9f-835e69870288: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService21"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::18"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_3b36f3f1-29d0-4d11-8b6b-ae2a89620d43/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::18"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="4a530305-da63-4a10-bd54-fea68adbdc01"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18962024-05-15T07:12:06.232ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8") failed: Property not found. retry in 286.166615ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8
18972024-05-15T07:12:06.262ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7b766ca5-5d0c-459d-bc67-f339bcac986b") failed: Property not found. retry in 124.811779ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7b766ca5-5d0c-459d-bc67-f339bcac986b
18982024-05-15T07:12:06.300ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_03e9085d-76ce-4798-9d35-12ceecd4c607") failed: Property not found. retry in 288.701945ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_03e9085d-76ce-4798-9d35-12ceecd4c607
18992024-05-15T07:12:06.301ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e
19002024-05-15T07:12:06.327ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_ae891598-60c4-47ca-bd0b-302a9e60a0bc: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService15"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::12"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_4e6ef30f-a7d9-4acc-99aa-59ae1084fa14/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::12"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="0f7f620d-0605-4161-b644-f2b21242487b"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19012024-05-15T07:12:06.377ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_42687f02-f9e0-4e2e-8b1e-8fcc2648d21d") failed: Property not found. retry in 249.772146ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_42687f02-f9e0-4e2e-8b1e-8fcc2648d21d
19022024-05-15T07:12:06.465ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054") failed: Property not found. retry in 36.638939ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054
19032024-05-15T07:12:06.579ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8") failed: Property not found. retry in 67.040375ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8
19042024-05-15T07:12:06.610ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d2e82ff1-e608-4c9b-8b24-06c6a00e8c2a") failed: Property not found. retry in 48.163472ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d2e82ff1-e608-4c9b-8b24-06c6a00e8c2a
19052024-05-15T07:12:06.610ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_ecac0828-7786-4503-af9f-835e69870288
19062024-05-15T07:12:06.659ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e0bae9f0-075a-41e1-a5e1-22133ae05550") failed: Property not found. retry in 61.501736ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e0bae9f0-075a-41e1-a5e1-22133ae05550
19072024-05-15T07:12:06.691ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8") failed: Property not found. retry in 490.920959ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8
19082024-05-15T07:12:06.723ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7b766ca5-5d0c-459d-bc67-f339bcac986b") failed: Property not found. retry in 285.551762ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7b766ca5-5d0c-459d-bc67-f339bcac986b
19092024-05-15T07:12:06.752ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_03e9085d-76ce-4798-9d35-12ceecd4c607") failed: Property not found. retry in 321.40866ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_03e9085d-76ce-4798-9d35-12ceecd4c607
19102024-05-15T07:12:06.753ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_ae891598-60c4-47ca-bd0b-302a9e60a0bc
19112024-05-15T07:12:06.804ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_42687f02-f9e0-4e2e-8b1e-8fcc2648d21d") failed: Property not found. retry in 527.986927ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_42687f02-f9e0-4e2e-8b1e-8fcc2648d21d
19122024-05-15T07:12:06.844ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054") failed: Property not found. retry in 63.01472ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054
19132024-05-15T07:12:06.875ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b") failed: Property not found. retry in 31.057474ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b
19142024-05-15T07:12:06.924ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8") failed: Property not found. retry in 134.427809ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8
19152024-05-15T07:12:06.969ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d2e82ff1-e608-4c9b-8b24-06c6a00e8c2a") failed: Property not found. retry in 50.379648ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d2e82ff1-e608-4c9b-8b24-06c6a00e8c2a
19162024-05-15T07:12:07.021ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e0bae9f0-075a-41e1-a5e1-22133ae05550") failed: Property not found. retry in 91.018381ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e0bae9f0-075a-41e1-a5e1-22133ae05550
19172024-05-15T07:12:07.087ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7b766ca5-5d0c-459d-bc67-f339bcac986b") failed: Property not found. retry in 442.077016ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7b766ca5-5d0c-459d-bc67-f339bcac986b
19182024-05-15T07:12:07.102ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_03e9085d-76ce-4798-9d35-12ceecd4c607") failed: Property not found. retry in 828.884504ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_03e9085d-76ce-4798-9d35-12ceecd4c607
19192024-05-15T07:12:07.117ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e") failed: Property not found. retry in 40.856196ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e
19202024-05-15T07:12:07.134ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054") failed: Property not found. retry in 147.274322ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054
19212024-05-15T07:12:07.150ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b") failed: Property not found. retry in 61.250375ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b
19222024-05-15T07:12:07.166ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8") failed: Property not found. retry in 213.765742ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8
19232024-05-15T07:12:07.183ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d2e82ff1-e608-4c9b-8b24-06c6a00e8c2a") failed: Property not found. retry in 199.533726ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d2e82ff1-e608-4c9b-8b24-06c6a00e8c2a
19242024-05-15T07:12:07.197ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ecac0828-7786-4503-af9f-835e69870288") failed: Property not found. retry in 58.902693ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ecac0828-7786-4503-af9f-835e69870288
19252024-05-15T07:12:07.212ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e0bae9f0-075a-41e1-a5e1-22133ae05550") failed: Property not found. retry in 130.59219ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e0bae9f0-075a-41e1-a5e1-22133ae05550
19262024-05-15T07:12:07.225ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8") failed: Property not found. retry in 480.695523ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8
19272024-05-15T07:12:07.237ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e") failed: Property not found. retry in 79.090983ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e
19282024-05-15T07:12:07.276ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b") failed: Property not found. retry in 264.09353ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b
19292024-05-15T07:12:07.333ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ecac0828-7786-4503-af9f-835e69870288") failed: Property not found. retry in 88.294372ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ecac0828-7786-4503-af9f-835e69870288
19302024-05-15T07:12:07.333ZINFOSledAgent (ServiceManager): Profile for oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService10"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::d"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/oximeter"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="id" value="7d8b5024-88b0-4173-8867-4e833f05550e"/> <propval type="astring" name="address" value="[fd00:1122:3344:101::d]:12223"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19312024-05-15T07:12:07.347ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e") failed: Property not found. retry in 101.591147ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e
19322024-05-15T07:12:07.367ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ae891598-60c4-47ca-bd0b-302a9e60a0bc") failed: Property not found. retry in 29.728007ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ae891598-60c4-47ca-bd0b-302a9e60a0bc
19332024-05-15T07:12:07.391ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_42687f02-f9e0-4e2e-8b1e-8fcc2648d21d") failed: Property not found. retry in 843.951732ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_42687f02-f9e0-4e2e-8b1e-8fcc2648d21d
19342024-05-15T07:12:07.408ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054") failed: Property not found. retry in 240.012902ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054
19352024-05-15T07:12:07.442ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8") failed: Property not found. retry in 540.691348ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8
19362024-05-15T07:12:07.498ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d2e82ff1-e608-4c9b-8b24-06c6a00e8c2a") failed: Property not found. retry in 378.626418ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d2e82ff1-e608-4c9b-8b24-06c6a00e8c2a
19372024-05-15T07:12:07.529ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ecac0828-7786-4503-af9f-835e69870288") failed: Property not found. retry in 160.296603ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ecac0828-7786-4503-af9f-835e69870288
19382024-05-15T07:12:07.553ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e0bae9f0-075a-41e1-a5e1-22133ae05550") failed: Property not found. retry in 228.176797ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e0bae9f0-075a-41e1-a5e1-22133ae05550
19392024-05-15T07:12:07.554ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e
19402024-05-15T07:12:07.592ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7b766ca5-5d0c-459d-bc67-f339bcac986b") failed: Property not found. retry in 611.97574ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7b766ca5-5d0c-459d-bc67-f339bcac986b
19412024-05-15T07:12:07.626ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e") failed: Property not found. retry in 550.99456ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e
19422024-05-15T07:12:07.648ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ae891598-60c4-47ca-bd0b-302a9e60a0bc") failed: Property not found. retry in 127.664502ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ae891598-60c4-47ca-bd0b-302a9e60a0bc
19432024-05-15T07:12:07.669ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b") failed: Property not found. retry in 207.599915ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b
19442024-05-15T07:12:07.693ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054") failed: Property not found. retry in 800.725519ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054
19452024-05-15T07:12:07.713ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ecac0828-7786-4503-af9f-835e69870288") failed: Property not found. retry in 208.371936ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ecac0828-7786-4503-af9f-835e69870288
19462024-05-15T07:12:07.732ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8") failed: Property not found. retry in 1.063199578s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8
19472024-05-15T07:12:07.802ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ae891598-60c4-47ca-bd0b-302a9e60a0bc") failed: Property not found. retry in 145.467796ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ae891598-60c4-47ca-bd0b-302a9e60a0bc
19482024-05-15T07:12:07.819ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e0bae9f0-075a-41e1-a5e1-22133ae05550") failed: Property not found. retry in 1.050813951s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e0bae9f0-075a-41e1-a5e1-22133ae05550
19492024-05-15T07:12:07.926ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b") failed: Property not found. retry in 565.227514ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b
19502024-05-15T07:12:08.012ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e") failed: Property not found. retry in 28.217488ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e
19512024-05-15T07:12:08.040ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d2e82ff1-e608-4c9b-8b24-06c6a00e8c2a") failed: Property not found. retry in 444.229363ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d2e82ff1-e608-4c9b-8b24-06c6a00e8c2a
19522024-05-15T07:12:08.072ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ecac0828-7786-4503-af9f-835e69870288") failed: Property not found. retry in 838.698203ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ecac0828-7786-4503-af9f-835e69870288
19532024-05-15T07:12:08.093ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e") failed: Property not found. retry in 145.201977ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e
19542024-05-15T07:12:08.111ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_03e9085d-76ce-4798-9d35-12ceecd4c607") failed: Property not found. retry in 1.349334644s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_03e9085d-76ce-4798-9d35-12ceecd4c607
19552024-05-15T07:12:08.127ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ae891598-60c4-47ca-bd0b-302a9e60a0bc") failed: Property not found. retry in 251.399211ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ae891598-60c4-47ca-bd0b-302a9e60a0bc
19562024-05-15T07:12:08.145ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8") failed: Property not found. retry in 478.35567ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8
19572024-05-15T07:12:08.198ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e") failed: Property not found. retry in 859.331364ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e
19582024-05-15T07:12:08.224ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7b766ca5-5d0c-459d-bc67-f339bcac986b") failed: Property not found. retry in 1.063639161s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7b766ca5-5d0c-459d-bc67-f339bcac986b
19592024-05-15T07:12:08.253ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_42687f02-f9e0-4e2e-8b1e-8fcc2648d21d") failed: Property not found. retry in 1.039199175s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_42687f02-f9e0-4e2e-8b1e-8fcc2648d21d
19602024-05-15T07:12:08.271ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e") failed: Property not found. retry in 160.392533ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e
19612024-05-15T07:12:08.400ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ae891598-60c4-47ca-bd0b-302a9e60a0bc") failed: Property not found. retry in 721.282628ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ae891598-60c4-47ca-bd0b-302a9e60a0bc
19622024-05-15T07:12:08.497ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e") failed: Property not found. retry in 441.426171ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e
19632024-05-15T07:12:08.498ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_e5538333-fcad-480c-9181-aa7e94aaf834: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService26"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::9"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte3"/> <propval type="astring" name="gateway" value="172.30.1.1"/> <propval type="astring" name="ip" value="172.30.1.6"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::9]:5353"/> <propval type="astring" name="dns_address" value="172.30.1.6:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19642024-05-15T07:12:08.500ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService27"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::8"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte4"/> <propval type="astring" name="gateway" value="172.30.1.1"/> <propval type="astring" name="ip" value="172.30.1.5"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::8]:5353"/> <propval type="astring" name="dns_address" value="172.30.1.5:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19652024-05-15T07:12:08.521ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054") failed: Property not found. retry in 1.028210724s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054
19662024-05-15T07:12:08.539ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b") failed: Property not found. retry in 857.969589ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b
19672024-05-15T07:12:08.557ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d2e82ff1-e608-4c9b-8b24-06c6a00e8c2a") failed: Property not found. retry in 1.453556572s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d2e82ff1-e608-4c9b-8b24-06c6a00e8c2a
19682024-05-15T07:12:08.557ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_external_dns_e5538333-fcad-480c-9181-aa7e94aaf834
19692024-05-15T07:12:08.564ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25
19702024-05-15T07:12:08.645ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8") failed: Property not found. retry in 822.642841ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8
19712024-05-15T07:12:08.812ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8") failed: Property not found. retry in 1.015358733s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8
19722024-05-15T07:12:08.883ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5538333-fcad-480c-9181-aa7e94aaf834") failed: Property not found. retry in 44.48971ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5538333-fcad-480c-9181-aa7e94aaf834
19732024-05-15T07:12:08.897ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25") failed: Property not found. retry in 68.036118ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25
19742024-05-15T07:12:08.912ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e0bae9f0-075a-41e1-a5e1-22133ae05550") failed: Property not found. retry in 906.113497ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e0bae9f0-075a-41e1-a5e1-22133ae05550
19752024-05-15T07:12:08.930ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ecac0828-7786-4503-af9f-835e69870288") failed: Property not found. retry in 950.267308ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ecac0828-7786-4503-af9f-835e69870288
19762024-05-15T07:12:08.945ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5538333-fcad-480c-9181-aa7e94aaf834") failed: Property not found. retry in 95.580084ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5538333-fcad-480c-9181-aa7e94aaf834
19772024-05-15T07:12:08.963ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e") failed: Property not found. retry in 749.823705ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e
19782024-05-15T07:12:09.025ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25") failed: Property not found. retry in 53.944077ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25
19792024-05-15T07:12:09.056ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5538333-fcad-480c-9181-aa7e94aaf834") failed: Property not found. retry in 240.49042ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5538333-fcad-480c-9181-aa7e94aaf834
19802024-05-15T07:12:09.071ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e") failed: Property not found. retry in 1.151980058s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e
19812024-05-15T07:12:09.090ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25") failed: Property not found. retry in 233.125045ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25
19822024-05-15T07:12:09.137ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ae891598-60c4-47ca-bd0b-302a9e60a0bc") failed: Property not found. retry in 1.382385141s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ae891598-60c4-47ca-bd0b-302a9e60a0bc
19832024-05-15T07:12:09.331ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7b766ca5-5d0c-459d-bc67-f339bcac986b") failed: Property not found. retry in 997.517618ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7b766ca5-5d0c-459d-bc67-f339bcac986b
19842024-05-15T07:12:09.356ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5538333-fcad-480c-9181-aa7e94aaf834") failed: Property not found. retry in 443.243693ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5538333-fcad-480c-9181-aa7e94aaf834
19852024-05-15T07:12:09.383ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25") failed: Property not found. retry in 475.133711ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25
19862024-05-15T07:12:09.402ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_42687f02-f9e0-4e2e-8b1e-8fcc2648d21d") failed: Property not found. retry in 684.03519ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_42687f02-f9e0-4e2e-8b1e-8fcc2648d21d
19872024-05-15T07:12:09.418ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b") failed: Property not found. retry in 760.131343ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b
19882024-05-15T07:12:09.481ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_03e9085d-76ce-4798-9d35-12ceecd4c607") failed: Property not found. retry in 1.176744391s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_03e9085d-76ce-4798-9d35-12ceecd4c607
19892024-05-15T07:12:09.499ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8") failed: Property not found. retry in 925.496001ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8
19902024-05-15T07:12:09.566ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054") failed: Property not found. retry in 1.369378606s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054
19912024-05-15T07:12:09.692ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_9cebcc66-fa56-4e34-af63-2c7ac52ff959: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService25"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::b"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte2"/> <propval type="astring" name="gateway" value="172.30.2.1"/> <propval type="astring" name="ip" value="172.30.2.6"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19922024-05-15T07:12:09.693ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_99861af6-438d-4bef-ba93-632645b46d2f: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService24"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::a"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte1"/> <propval type="astring" name="gateway" value="172.30.2.1"/> <propval type="astring" name="ip" value="172.30.2.5"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19932024-05-15T07:12:09.693ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService28"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::c"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte5"/> <propval type="astring" name="gateway" value="172.30.2.1"/> <propval type="astring" name="ip" value="172.30.2.7"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19942024-05-15T07:12:09.693ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_nexus_99861af6-438d-4bef-ba93-632645b46d2f
19952024-05-15T07:12:09.703ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_nexus_9cebcc66-fa56-4e34-af63-2c7ac52ff959
19962024-05-15T07:12:09.711ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_nexus_feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3
19972024-05-15T07:12:09.738ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e") failed: Property not found. retry in 1.372029068s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e
19982024-05-15T07:12:09.822ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5538333-fcad-480c-9181-aa7e94aaf834") failed: Property not found. retry in 401.029705ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5538333-fcad-480c-9181-aa7e94aaf834
19992024-05-15T07:12:09.840ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e0bae9f0-075a-41e1-a5e1-22133ae05550") failed: Property not found. retry in 1.172856053s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e0bae9f0-075a-41e1-a5e1-22133ae05550
20002024-05-15T07:12:09.861ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8") failed: Property not found. retry in 933.29296ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8
20012024-05-15T07:12:09.891ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25") failed: Property not found. retry in 789.186134ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25
20022024-05-15T07:12:09.935ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ecac0828-7786-4503-af9f-835e69870288") failed: Property not found. retry in 809.355217ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ecac0828-7786-4503-af9f-835e69870288
20032024-05-15T07:12:10.037ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d2e82ff1-e608-4c9b-8b24-06c6a00e8c2a") failed: Property not found. retry in 1.082991259s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d2e82ff1-e608-4c9b-8b24-06c6a00e8c2a
20042024-05-15T07:12:10.079ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_99861af6-438d-4bef-ba93-632645b46d2f") failed: Property not found. retry in 56.129409ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_99861af6-438d-4bef-ba93-632645b46d2f
20052024-05-15T07:12:10.107ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_42687f02-f9e0-4e2e-8b1e-8fcc2648d21d") failed: Property not found. retry in 1.162919272s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_42687f02-f9e0-4e2e-8b1e-8fcc2648d21d
20062024-05-15T07:12:10.122ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9cebcc66-fa56-4e34-af63-2c7ac52ff959") failed: Property not found. retry in 48.491294ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9cebcc66-fa56-4e34-af63-2c7ac52ff959
20072024-05-15T07:12:10.138ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3") failed: Property not found. retry in 60.223385ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3
20082024-05-15T07:12:10.254ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_99861af6-438d-4bef-ba93-632645b46d2f") failed: Property not found. retry in 54.990004ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_99861af6-438d-4bef-ba93-632645b46d2f
20092024-05-15T07:12:10.472ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5538333-fcad-480c-9181-aa7e94aaf834") failed: Property not found. retry in 1.433055217s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5538333-fcad-480c-9181-aa7e94aaf834
20102024-05-15T07:12:10.531ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7b766ca5-5d0c-459d-bc67-f339bcac986b") failed: Property not found. retry in 1.486001911s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7b766ca5-5d0c-459d-bc67-f339bcac986b
20112024-05-15T07:12:10.587ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e") failed: Property not found. retry in 1.364949824s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e
20122024-05-15T07:12:10.609ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_99861af6-438d-4bef-ba93-632645b46d2f") failed: Property not found. retry in 108.476781ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_99861af6-438d-4bef-ba93-632645b46d2f
20132024-05-15T07:12:10.650ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9cebcc66-fa56-4e34-af63-2c7ac52ff959") failed: Property not found. retry in 52.780146ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9cebcc66-fa56-4e34-af63-2c7ac52ff959
20142024-05-15T07:12:10.667ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ae891598-60c4-47ca-bd0b-302a9e60a0bc") failed: Property not found. retry in 680.299636ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ae891598-60c4-47ca-bd0b-302a9e60a0bc
20152024-05-15T07:12:10.683ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b") failed: Property not found. retry in 968.209085ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b
20162024-05-15T07:12:10.698ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8") failed: Property not found. retry in 645.662698ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8
20172024-05-15T07:12:10.715ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3") failed: Property not found. retry in 59.522041ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3
20182024-05-15T07:12:10.731ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25") failed: Property not found. retry in 1.090361916s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25
20192024-05-15T07:12:10.746ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_03e9085d-76ce-4798-9d35-12ceecd4c607") failed: Property not found. retry in 853.412333ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_03e9085d-76ce-4798-9d35-12ceecd4c607
20202024-05-15T07:12:10.761ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_99861af6-438d-4bef-ba93-632645b46d2f") failed: Property not found. retry in 286.011784ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_99861af6-438d-4bef-ba93-632645b46d2f
20212024-05-15T07:12:10.775ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9cebcc66-fa56-4e34-af63-2c7ac52ff959") failed: Property not found. retry in 279.648672ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9cebcc66-fa56-4e34-af63-2c7ac52ff959
20222024-05-15T07:12:10.790ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ecac0828-7786-4503-af9f-835e69870288") failed: Property not found. retry in 927.985831ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ecac0828-7786-4503-af9f-835e69870288
20232024-05-15T07:12:10.804ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3") failed: Property not found. retry in 297.862206ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3
20242024-05-15T07:12:10.818ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8") failed: Property not found. retry in 860.700389ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8
20252024-05-15T07:12:10.955ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054") failed: Property not found. retry in 914.660657ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054
20262024-05-15T07:12:11.027ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e0bae9f0-075a-41e1-a5e1-22133ae05550") failed: Property not found. retry in 1.255970183s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e0bae9f0-075a-41e1-a5e1-22133ae05550
20272024-05-15T07:12:11.061ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_99861af6-438d-4bef-ba93-632645b46d2f") failed: Property not found. retry in 1.073331655s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_99861af6-438d-4bef-ba93-632645b46d2f
20282024-05-15T07:12:11.074ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9cebcc66-fa56-4e34-af63-2c7ac52ff959") failed: Property not found. retry in 548.942711ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9cebcc66-fa56-4e34-af63-2c7ac52ff959
20292024-05-15T07:12:11.114ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3") failed: Property not found. retry in 329.540523ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3
20302024-05-15T07:12:11.125ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e") failed: Property not found. retry in 881.507601ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e
20312024-05-15T07:12:11.136ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d2e82ff1-e608-4c9b-8b24-06c6a00e8c2a") failed: Property not found. retry in 624.614174ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d2e82ff1-e608-4c9b-8b24-06c6a00e8c2a
20322024-05-15T07:12:11.282ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_42687f02-f9e0-4e2e-8b1e-8fcc2648d21d") failed: Property not found. retry in 1.022325299s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_42687f02-f9e0-4e2e-8b1e-8fcc2648d21d
20332024-05-15T07:12:11.357ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8") failed: Property not found. retry in 513.093397ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8
20342024-05-15T07:12:11.369ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ae891598-60c4-47ca-bd0b-302a9e60a0bc") failed: Property not found. retry in 1.050707225s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ae891598-60c4-47ca-bd0b-302a9e60a0bc
20352024-05-15T07:12:11.457ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3") failed: Property not found. retry in 709.201981ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3
20362024-05-15T07:12:11.613ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_03e9085d-76ce-4798-9d35-12ceecd4c607") failed: Property not found. retry in 1.347231978s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_03e9085d-76ce-4798-9d35-12ceecd4c607
20372024-05-15T07:12:11.635ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9cebcc66-fa56-4e34-af63-2c7ac52ff959") failed: Property not found. retry in 776.032126ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9cebcc66-fa56-4e34-af63-2c7ac52ff959
20382024-05-15T07:12:11.663ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b") failed: Property not found. retry in 1.234694938s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b
20392024-05-15T07:12:11.693ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8") failed: Property not found. retry in 629.795621ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8
20402024-05-15T07:12:11.730ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ecac0828-7786-4503-af9f-835e69870288") failed: Property not found. retry in 1.041026489s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ecac0828-7786-4503-af9f-835e69870288
20412024-05-15T07:12:11.772ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d2e82ff1-e608-4c9b-8b24-06c6a00e8c2a") failed: Property not found. retry in 643.569305ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d2e82ff1-e608-4c9b-8b24-06c6a00e8c2a
20422024-05-15T07:12:11.834ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25") failed: Property not found. retry in 1.110329784s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25
20432024-05-15T07:12:11.881ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054") failed: Property not found. retry in 738.081824ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054
20442024-05-15T07:12:11.893ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8") failed: Property not found. retry in 586.745915ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8
20452024-05-15T07:12:11.916ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5538333-fcad-480c-9181-aa7e94aaf834") failed: Property not found. retry in 1.328108368s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5538333-fcad-480c-9181-aa7e94aaf834
20462024-05-15T07:12:11.963ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e") failed: Property not found. retry in 616.691972ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e
20472024-05-15T07:12:12.019ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e") failed: Property not found. retry in 1.378920914s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e
20482024-05-15T07:12:12.029ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7b766ca5-5d0c-459d-bc67-f339bcac986b") failed: Property not found. retry in 1.062282019s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7b766ca5-5d0c-459d-bc67-f339bcac986b
20492024-05-15T07:12:12.152ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_99861af6-438d-4bef-ba93-632645b46d2f") failed: Property not found. retry in 589.245503ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_99861af6-438d-4bef-ba93-632645b46d2f
20502024-05-15T07:12:12.179ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3") failed: Property not found. retry in 1.415786126s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3
20512024-05-15T07:12:12.297ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e0bae9f0-075a-41e1-a5e1-22133ae05550") failed: Property not found. retry in 1.013116895s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e0bae9f0-075a-41e1-a5e1-22133ae05550
20522024-05-15T07:12:12.373ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8") failed: Property not found. retry in 539.704119ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8
20532024-05-15T07:12:12.385ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_42687f02-f9e0-4e2e-8b1e-8fcc2648d21d") failed: Property not found. retry in 901.398021ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_42687f02-f9e0-4e2e-8b1e-8fcc2648d21d
20542024-05-15T07:12:12.436ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9cebcc66-fa56-4e34-af63-2c7ac52ff959") failed: Property not found. retry in 1.033857451s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9cebcc66-fa56-4e34-af63-2c7ac52ff959
20552024-05-15T07:12:12.447ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ae891598-60c4-47ca-bd0b-302a9e60a0bc") failed: Property not found. retry in 1.082133849s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ae891598-60c4-47ca-bd0b-302a9e60a0bc
20562024-05-15T07:12:12.458ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d2e82ff1-e608-4c9b-8b24-06c6a00e8c2a") failed: Property not found. retry in 1.385040595s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d2e82ff1-e608-4c9b-8b24-06c6a00e8c2a
20572024-05-15T07:12:12.492ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8") failed: Property not found. retry in 603.732884ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8
20582024-05-15T07:12:12.596ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e") failed: Property not found. retry in 546.905954ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e
20592024-05-15T07:12:12.640ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054") failed: Property not found. retry in 798.19698ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054
20602024-05-15T07:12:12.652ZINFOSledAgent (ServiceManager): Profile for oxz_clickhouse_7ce930b4-1bea-41ce-8c9a-2aa16fd16d05: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService23"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::e"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::e"/> <propval type="astring" name="listen_port" value="8123"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
20612024-05-15T07:12:12.653ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_clickhouse_7ce930b4-1bea-41ce-8c9a-2aa16fd16d05
20622024-05-15T07:12:12.756ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_99861af6-438d-4bef-ba93-632645b46d2f") failed: Property not found. retry in 951.509426ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_99861af6-438d-4bef-ba93-632645b46d2f
20632024-05-15T07:12:12.784ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ecac0828-7786-4503-af9f-835e69870288") failed: Property not found. retry in 746.175243ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ecac0828-7786-4503-af9f-835e69870288
20642024-05-15T07:12:12.899ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7ce930b4-1bea-41ce-8c9a-2aa16fd16d05") failed: Property not found. retry in 58.561598ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7ce930b4-1bea-41ce-8c9a-2aa16fd16d05
20652024-05-15T07:12:12.914ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b") failed: Property not found. retry in 708.802853ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b
20662024-05-15T07:12:12.927ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8") failed: Property not found. retry in 1.432240083s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8
20672024-05-15T07:12:12.959ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25") failed: Property not found. retry in 718.649724ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25
20682024-05-15T07:12:12.971ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7ce930b4-1bea-41ce-8c9a-2aa16fd16d05") failed: Property not found. retry in 116.051063ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7ce930b4-1bea-41ce-8c9a-2aa16fd16d05
20692024-05-15T07:12:12.985ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_03e9085d-76ce-4798-9d35-12ceecd4c607") failed: Property not found. retry in 696.004218ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_03e9085d-76ce-4798-9d35-12ceecd4c607
20702024-05-15T07:12:13.122ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7ce930b4-1bea-41ce-8c9a-2aa16fd16d05") failed: Property not found. retry in 111.495997ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7ce930b4-1bea-41ce-8c9a-2aa16fd16d05
20712024-05-15T07:12:13.164ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7b766ca5-5d0c-459d-bc67-f339bcac986b") failed: Property not found. retry in 1.186057807s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7b766ca5-5d0c-459d-bc67-f339bcac986b
20722024-05-15T07:12:13.207ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8") failed: Property not found. retry in 1.15870666s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8
20732024-05-15T07:12:13.222ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e") failed: Property not found. retry in 802.699278ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e
20742024-05-15T07:12:13.332ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7ce930b4-1bea-41ce-8c9a-2aa16fd16d05") failed: Property not found. retry in 504.484904ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7ce930b4-1bea-41ce-8c9a-2aa16fd16d05
20752024-05-15T07:12:13.341ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 86.809086607s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:426 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
20762024-05-15T07:12:13.349ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5538333-fcad-480c-9181-aa7e94aaf834") failed: Property not found. retry in 1.423001353s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5538333-fcad-480c-9181-aa7e94aaf834
20772024-05-15T07:12:13.363ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_42687f02-f9e0-4e2e-8b1e-8fcc2648d21d") failed: Property not found. retry in 658.339003ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_42687f02-f9e0-4e2e-8b1e-8fcc2648d21d
20782024-05-15T07:12:13.376ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e0bae9f0-075a-41e1-a5e1-22133ae05550") failed: Property not found. retry in 1.371182126s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e0bae9f0-075a-41e1-a5e1-22133ae05550
20792024-05-15T07:12:13.411ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e") failed: Property not found. retry in 1.150556605s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e
20802024-05-15T07:12:13.452ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054") failed: Property not found. retry in 1.149615517s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054
20812024-05-15T07:12:13.480ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9cebcc66-fa56-4e34-af63-2c7ac52ff959") failed: Property not found. retry in 631.371466ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9cebcc66-fa56-4e34-af63-2c7ac52ff959
20822024-05-15T07:12:13.540ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ae891598-60c4-47ca-bd0b-302a9e60a0bc") failed: Property not found. retry in 1.403064319s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ae891598-60c4-47ca-bd0b-302a9e60a0bc
20832024-05-15T07:12:13.551ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ecac0828-7786-4503-af9f-835e69870288") failed: Property not found. retry in 920.807947ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ecac0828-7786-4503-af9f-835e69870288
20842024-05-15T07:12:13.605ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3") failed: Property not found. retry in 1.058413597s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3
20852024-05-15T07:12:13.633ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b") failed: Property not found. retry in 864.159365ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b
20862024-05-15T07:12:13.689ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25") failed: Property not found. retry in 1.277260488s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25
20872024-05-15T07:12:13.700ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_03e9085d-76ce-4798-9d35-12ceecd4c607") failed: Property not found. retry in 1.161489836s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_03e9085d-76ce-4798-9d35-12ceecd4c607
20882024-05-15T07:12:13.719ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_99861af6-438d-4bef-ba93-632645b46d2f") failed: Property not found. retry in 558.418778ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_99861af6-438d-4bef-ba93-632645b46d2f
20892024-05-15T07:12:13.849ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7ce930b4-1bea-41ce-8c9a-2aa16fd16d05") failed: Property not found. retry in 1.053623966s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7ce930b4-1bea-41ce-8c9a-2aa16fd16d05
20902024-05-15T07:12:13.863ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d2e82ff1-e608-4c9b-8b24-06c6a00e8c2a") failed: Property not found. retry in 1.308004939s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d2e82ff1-e608-4c9b-8b24-06c6a00e8c2a
20912024-05-15T07:12:14.039ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_42687f02-f9e0-4e2e-8b1e-8fcc2648d21d") failed: Property not found. retry in 1.179343084s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_42687f02-f9e0-4e2e-8b1e-8fcc2648d21d
20922024-05-15T07:12:14.055ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e") failed: Property not found. retry in 845.634618ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e
20932024-05-15T07:12:14.127ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9cebcc66-fa56-4e34-af63-2c7ac52ff959") failed: Property not found. retry in 991.24235ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9cebcc66-fa56-4e34-af63-2c7ac52ff959
20942024-05-15T07:12:14.304ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_99861af6-438d-4bef-ba93-632645b46d2f") failed: Property not found. retry in 906.538841ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_99861af6-438d-4bef-ba93-632645b46d2f
20952024-05-15T07:12:14.377ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7b766ca5-5d0c-459d-bc67-f339bcac986b") failed: Property not found. retry in 1.152650042s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7b766ca5-5d0c-459d-bc67-f339bcac986b
20962024-05-15T07:12:14.400ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8") failed: Property not found. retry in 629.000463ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8
20972024-05-15T07:12:14.428ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8") failed: Property not found. retry in 1.40248341s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8
20982024-05-15T07:12:14.494ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ecac0828-7786-4503-af9f-835e69870288") failed: Property not found. retry in 1.464903022s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ecac0828-7786-4503-af9f-835e69870288
20992024-05-15T07:12:14.538ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b") failed: Property not found. retry in 643.901069ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b
21002024-05-15T07:12:14.584ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e") failed: Property not found. retry in 1.090909125s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e
21012024-05-15T07:12:14.617ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054") failed: Property not found. retry in 1.043662349s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054
21022024-05-15T07:12:14.682ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3") failed: Property not found. retry in 1.008319442s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3
21032024-05-15T07:12:14.766ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e0bae9f0-075a-41e1-a5e1-22133ae05550") failed: Property not found. retry in 1.282779159s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e0bae9f0-075a-41e1-a5e1-22133ae05550
21042024-05-15T07:12:14.796ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5538333-fcad-480c-9181-aa7e94aaf834") failed: Property not found. retry in 636.933288ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5538333-fcad-480c-9181-aa7e94aaf834
21052024-05-15T07:12:14.880ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_03e9085d-76ce-4798-9d35-12ceecd4c607") failed: Property not found. retry in 1.022063988s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_03e9085d-76ce-4798-9d35-12ceecd4c607
21062024-05-15T07:12:14.922ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e") failed: Property not found. retry in 1.150423272s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e
21072024-05-15T07:12:14.942ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7ce930b4-1bea-41ce-8c9a-2aa16fd16d05") failed: Property not found. retry in 1.453159607s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7ce930b4-1bea-41ce-8c9a-2aa16fd16d05
21082024-05-15T07:12:14.962ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ae891598-60c4-47ca-bd0b-302a9e60a0bc") failed: Property not found. retry in 708.472654ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ae891598-60c4-47ca-bd0b-302a9e60a0bc
21092024-05-15T07:12:14.984ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25") failed: Property not found. retry in 759.783325ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25
21102024-05-15T07:12:15.044ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8") failed: Property not found. retry in 528.256604ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8
21112024-05-15T07:12:15.133ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9cebcc66-fa56-4e34-af63-2c7ac52ff959") failed: Property not found. retry in 1.417235023s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9cebcc66-fa56-4e34-af63-2c7ac52ff959
21122024-05-15T07:12:15.191ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d2e82ff1-e608-4c9b-8b24-06c6a00e8c2a") failed: Property not found. retry in 990.520707ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d2e82ff1-e608-4c9b-8b24-06c6a00e8c2a
21132024-05-15T07:12:15.205ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b") failed: Property not found. retry in 1.472801124s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b
21142024-05-15T07:12:15.222ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_99861af6-438d-4bef-ba93-632645b46d2f") failed: Property not found. retry in 797.24176ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_99861af6-438d-4bef-ba93-632645b46d2f
21152024-05-15T07:12:15.236ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_42687f02-f9e0-4e2e-8b1e-8fcc2648d21d") failed: Property not found. retry in 1.496639761s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_42687f02-f9e0-4e2e-8b1e-8fcc2648d21d
21162024-05-15T07:12:15.450ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5538333-fcad-480c-9181-aa7e94aaf834") failed: Property not found. retry in 1.353067187s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5538333-fcad-480c-9181-aa7e94aaf834
21172024-05-15T07:12:15.543ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7b766ca5-5d0c-459d-bc67-f339bcac986b") failed: Property not found. retry in 1.159405902s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7b766ca5-5d0c-459d-bc67-f339bcac986b
21182024-05-15T07:12:15.590ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8") failed: Property not found. retry in 915.292997ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8
21192024-05-15T07:12:15.675ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054") failed: Property not found. retry in 734.952931ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054
21202024-05-15T07:12:15.689ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ae891598-60c4-47ca-bd0b-302a9e60a0bc") failed: Property not found. retry in 844.183427ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ae891598-60c4-47ca-bd0b-302a9e60a0bc
21212024-05-15T07:12:15.703ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e") failed: Property not found. retry in 1.439051992s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e
21222024-05-15T07:12:15.718ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3") failed: Property not found. retry in 1.497972851s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3
21232024-05-15T07:12:15.756ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25") failed: Property not found. retry in 1.427882038s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25
21242024-05-15T07:12:15.841ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8") failed: Property not found. retry in 1.331791726s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8
21252024-05-15T07:12:15.916ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_03e9085d-76ce-4798-9d35-12ceecd4c607") failed: Property not found. retry in 1.304453859s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_03e9085d-76ce-4798-9d35-12ceecd4c607
21262024-05-15T07:12:15.974ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ecac0828-7786-4503-af9f-835e69870288") failed: Property not found. retry in 991.347004ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ecac0828-7786-4503-af9f-835e69870288
21272024-05-15T07:12:16.031ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_99861af6-438d-4bef-ba93-632645b46d2f") failed: Property not found. retry in 1.437009393s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_99861af6-438d-4bef-ba93-632645b46d2f
21282024-05-15T07:12:16.059ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e0bae9f0-075a-41e1-a5e1-22133ae05550") failed: Property not found. retry in 1.428543717s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e0bae9f0-075a-41e1-a5e1-22133ae05550
21292024-05-15T07:12:16.088ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e") failed: Property not found. retry in 615.056511ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e
21302024-05-15T07:12:16.197ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d2e82ff1-e608-4c9b-8b24-06c6a00e8c2a") failed: Property not found. retry in 1.216011795s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d2e82ff1-e608-4c9b-8b24-06c6a00e8c2a
21312024-05-15T07:12:16.409ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7ce930b4-1bea-41ce-8c9a-2aa16fd16d05") failed: Property not found. retry in 643.465493ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7ce930b4-1bea-41ce-8c9a-2aa16fd16d05
21322024-05-15T07:12:16.421ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054") failed: Property not found. retry in 977.222651ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054
21332024-05-15T07:12:16.520ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8") failed: Property not found. retry in 706.421811ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8
21342024-05-15T07:12:16.545ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ae891598-60c4-47ca-bd0b-302a9e60a0bc") failed: Property not found. retry in 818.545068ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ae891598-60c4-47ca-bd0b-302a9e60a0bc
21352024-05-15T07:12:16.569ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9cebcc66-fa56-4e34-af63-2c7ac52ff959") failed: Property not found. retry in 1.045946717s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9cebcc66-fa56-4e34-af63-2c7ac52ff959
21362024-05-15T07:12:16.691ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b") failed: Property not found. retry in 1.102909961s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b
21372024-05-15T07:12:16.715ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7b766ca5-5d0c-459d-bc67-f339bcac986b") failed: Property not found. retry in 614.670761ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7b766ca5-5d0c-459d-bc67-f339bcac986b
21382024-05-15T07:12:16.727ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e") failed: Property not found. retry in 889.513972ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e
21392024-05-15T07:12:16.745ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_42687f02-f9e0-4e2e-8b1e-8fcc2648d21d") failed: Property not found. retry in 828.220259ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_42687f02-f9e0-4e2e-8b1e-8fcc2648d21d
21402024-05-15T07:12:16.816ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5538333-fcad-480c-9181-aa7e94aaf834") failed: Property not found. retry in 948.520622ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5538333-fcad-480c-9181-aa7e94aaf834
21412024-05-15T07:12:16.977ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ecac0828-7786-4503-af9f-835e69870288") failed: Property not found. retry in 582.662369ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ecac0828-7786-4503-af9f-835e69870288
21422024-05-15T07:12:17.067ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7ce930b4-1bea-41ce-8c9a-2aa16fd16d05") failed: Property not found. retry in 943.743124ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7ce930b4-1bea-41ce-8c9a-2aa16fd16d05
21432024-05-15T07:12:17.158ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e") failed: Property not found. retry in 1.032182523s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e
21442024-05-15T07:12:17.190ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8") failed: Property not found. retry in 1.036847359s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8
21452024-05-15T07:12:17.202ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25") failed: Property not found. retry in 613.853296ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25
21462024-05-15T07:12:17.228ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3") failed: Property not found. retry in 1.164717044s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3
21472024-05-15T07:12:17.379ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ae891598-60c4-47ca-bd0b-302a9e60a0bc") failed: Property not found. retry in 1.442480685s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ae891598-60c4-47ca-bd0b-302a9e60a0bc
21482024-05-15T07:12:17.413ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054") failed: Property not found. retry in 584.773164ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054
21492024-05-15T07:12:17.481ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_99861af6-438d-4bef-ba93-632645b46d2f") failed: Property not found. retry in 1.248568006s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_99861af6-438d-4bef-ba93-632645b46d2f
21502024-05-15T07:12:17.499ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e0bae9f0-075a-41e1-a5e1-22133ae05550") failed: Property not found. retry in 629.774355ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e0bae9f0-075a-41e1-a5e1-22133ae05550
21512024-05-15T07:12:17.571ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ecac0828-7786-4503-af9f-835e69870288") failed: Property not found. retry in 642.826381ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ecac0828-7786-4503-af9f-835e69870288
21522024-05-15T07:12:17.628ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9cebcc66-fa56-4e34-af63-2c7ac52ff959") failed: Property not found. retry in 846.772815ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9cebcc66-fa56-4e34-af63-2c7ac52ff959
21532024-05-15T07:12:17.640ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e") failed: Property not found. retry in 543.4799ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e
21542024-05-15T07:12:17.780ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5538333-fcad-480c-9181-aa7e94aaf834") failed: Property not found. retry in 1.391797863s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5538333-fcad-480c-9181-aa7e94aaf834
21552024-05-15T07:12:17.809ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b") failed: Property not found. retry in 771.575832ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b
21562024-05-15T07:12:17.831ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25") failed: Property not found. retry in 503.859158ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25
21572024-05-15T07:12:18.010ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054") failed: Property not found. retry in 1.287753794s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054
21582024-05-15T07:12:18.034ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7ce930b4-1bea-41ce-8c9a-2aa16fd16d05") failed: Property not found. retry in 1.189568454s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7ce930b4-1bea-41ce-8c9a-2aa16fd16d05
21592024-05-15T07:12:18.146ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e0bae9f0-075a-41e1-a5e1-22133ae05550") failed: Property not found. retry in 675.214636ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e0bae9f0-075a-41e1-a5e1-22133ae05550
21602024-05-15T07:12:18.196ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e") failed: Property not found. retry in 873.506177ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e
21612024-05-15T07:12:18.207ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e") failed: Property not found. retry in 518.999038ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e
21622024-05-15T07:12:18.228ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ecac0828-7786-4503-af9f-835e69870288") failed: Property not found. retry in 502.924566ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ecac0828-7786-4503-af9f-835e69870288
21632024-05-15T07:12:18.250ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8") failed: Property not found. retry in 761.042156ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8
21642024-05-15T07:12:18.352ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25") failed: Property not found. retry in 1.487241809s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25
21652024-05-15T07:12:18.406ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3") failed: Property not found. retry in 1.449707724s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3
21662024-05-15T07:12:18.492ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9cebcc66-fa56-4e34-af63-2c7ac52ff959") failed: Property not found. retry in 1.279970867s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9cebcc66-fa56-4e34-af63-2c7ac52ff959
21672024-05-15T07:12:18.595ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b") failed: Property not found. retry in 741.054291ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b
21682024-05-15T07:12:18.743ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e") failed: Property not found. retry in 979.168261ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e
21692024-05-15T07:12:18.761ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_99861af6-438d-4bef-ba93-632645b46d2f") failed: Property not found. retry in 1.218236171s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_99861af6-438d-4bef-ba93-632645b46d2f
21702024-05-15T07:12:18.775ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ecac0828-7786-4503-af9f-835e69870288") failed: Property not found. retry in 995.316115ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ecac0828-7786-4503-af9f-835e69870288
21712024-05-15T07:12:18.837ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ae891598-60c4-47ca-bd0b-302a9e60a0bc") failed: Property not found. retry in 697.991814ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ae891598-60c4-47ca-bd0b-302a9e60a0bc
21722024-05-15T07:12:18.850ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e0bae9f0-075a-41e1-a5e1-22133ae05550") failed: Property not found. retry in 1.24416045s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e0bae9f0-075a-41e1-a5e1-22133ae05550
21732024-05-15T07:12:19.189ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5538333-fcad-480c-9181-aa7e94aaf834") failed: Property not found. retry in 779.848844ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5538333-fcad-480c-9181-aa7e94aaf834
21742024-05-15T07:12:19.242ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7ce930b4-1bea-41ce-8c9a-2aa16fd16d05") failed: Property not found. retry in 1.006240209s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7ce930b4-1bea-41ce-8c9a-2aa16fd16d05
21752024-05-15T07:12:19.739ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e") failed: Property not found. retry in 723.503383ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e
21762024-05-15T07:12:19.807ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9cebcc66-fa56-4e34-af63-2c7ac52ff959") failed: Property not found. retry in 1.446870095s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9cebcc66-fa56-4e34-af63-2c7ac52ff959
21772024-05-15T07:12:19.852ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25") failed: Property not found. retry in 504.748212ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25
21782024-05-15T07:12:19.868ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3") failed: Property not found. retry in 747.188043ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3
21792024-05-15T07:12:19.981ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5538333-fcad-480c-9181-aa7e94aaf834") failed: Property not found. retry in 1.092358054s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5538333-fcad-480c-9181-aa7e94aaf834
21802024-05-15T07:12:19.993ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_99861af6-438d-4bef-ba93-632645b46d2f") failed: Property not found. retry in 718.369455ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_99861af6-438d-4bef-ba93-632645b46d2f
21812024-05-15T07:12:20.260ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7ce930b4-1bea-41ce-8c9a-2aa16fd16d05") failed: Property not found. retry in 1.399675741s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7ce930b4-1bea-41ce-8c9a-2aa16fd16d05
21822024-05-15T07:12:20.368ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25") failed: Property not found. retry in 872.868086ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25
21832024-05-15T07:12:20.473ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e") failed: Property not found. retry in 1.319788345s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e
21842024-05-15T07:12:20.631ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3") failed: Property not found. retry in 1.225429317s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3
21852024-05-15T07:12:20.725ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_99861af6-438d-4bef-ba93-632645b46d2f") failed: Property not found. retry in 1.141244141s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_99861af6-438d-4bef-ba93-632645b46d2f
21862024-05-15T07:12:21.086ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5538333-fcad-480c-9181-aa7e94aaf834") failed: Property not found. retry in 1.306120121s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5538333-fcad-480c-9181-aa7e94aaf834
21872024-05-15T07:12:21.252ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25") failed: Property not found. retry in 536.034198ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25
21882024-05-15T07:12:21.264ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9cebcc66-fa56-4e34-af63-2c7ac52ff959") failed: Property not found. retry in 817.142215ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9cebcc66-fa56-4e34-af63-2c7ac52ff959
21892024-05-15T07:12:21.678ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_7ce930b4-1bea-41ce-8c9a-2aa16fd16d05") failed: Property not found. retry in 1.088846718s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_7ce930b4-1bea-41ce-8c9a-2aa16fd16d05
21902024-05-15T07:12:22.811ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_external_dns_e5538333-fcad-480c-9181-aa7e94aaf834
21912024-05-15T07:12:22.811ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_external_dns_12908044-fccf-4c46-8e7c-c2a0be1bfb25
21922024-05-15T07:12:22.811ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_oximeter_7d8b5024-88b0-4173-8867-4e833f05550e
21932024-05-15T07:12:22.811ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_pantry_638e5dab-76c4-4b78-b15a-a7faf399d1d8
21942024-05-15T07:12:22.811ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_7b766ca5-5d0c-459d-bc67-f339bcac986b
21952024-05-15T07:12:22.811ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_03e9085d-76ce-4798-9d35-12ceecd4c607
21962024-05-15T07:12:22.811ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_d62b80e0-a2d7-4b05-8ee7-7f6a9516881e
21972024-05-15T07:12:22.811ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_nexus_99861af6-438d-4bef-ba93-632645b46d2f
21982024-05-15T07:12:22.811ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_nexus_9cebcc66-fa56-4e34-af63-2c7ac52ff959
21992024-05-15T07:12:22.811ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_ae891598-60c4-47ca-bd0b-302a9e60a0bc
22002024-05-15T07:12:22.811ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_pantry_42687f02-f9e0-4e2e-8b1e-8fcc2648d21d
22012024-05-15T07:12:22.811ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_d41b4327-a68c-4bde-a48b-cfcb75b4c054
22022024-05-15T07:12:22.813ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_eece7197-e5eb-4531-85f7-6700f07faf6b
22032024-05-15T07:12:22.813ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_c06cd1db-bb07-41fa-af89-0fd52ff0a0a8
22042024-05-15T07:12:22.813ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_pantry_d2e82ff1-e608-4c9b-8b24-06c6a00e8c2a
22052024-05-15T07:12:22.813ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_ecac0828-7786-4503-af9f-835e69870288
22062024-05-15T07:12:22.813ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_e0bae9f0-075a-41e1-a5e1-22133ae05550
22072024-05-15T07:12:22.814ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_clickhouse_7ce930b4-1bea-41ce-8c9a-2aa16fd16d05
22082024-05-15T07:12:22.814ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_nexus_feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3
22092024-05-15T07:12:22.823ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
22102024-05-15T07:12:22.825ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
22112024-05-15T07:12:22.834ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 24698311 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:60075 req_id = c072f072-967b-4315-9f5d-017448974204 response_code = 204 uri = /omicron-zones
22122024-05-15T07:12:22.867ZINFOSledAgent (RSS): Finished setting up services file = sled-agent/src/rack_setup/service.rs:1236
22132024-05-15T07:12:22.868ZINFOSledAgent (RSS): Writing ledger to /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
22142024-05-15T07:12:22.868ZINFOSledAgent (RSS): Writing ledger to /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
22152024-05-15T07:12:22.881ZINFOSledAgent (RSS): Handing off control to Nexus file = sled-agent/src/rack_setup/service.rs:680
22162024-05-15T07:12:22.894ZINFOSledAgent (RSS): Nexus address: [fd00:1122:3344:101::a]:12221 file = sled-agent/src/rack_setup/service.rs:693
22172024-05-15T07:12:22.974ZINFOSledAgent (RSS): rack_network_config: RackNetworkConfigV1 { bfd: [], bgp: [], infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV1 { addresses: [ V4( Ipv4Network { addr: 10.151.2.174, prefix: 24, }, ), ], autoneg: false, bgp_peers: [], port: "qsfp0", routes: [ RouteConfig { destination: V4( Ipv4Network { addr: 0.0.0.0, prefix: 0, }, ), nexthop: 10.151.2.100, vlan_id: None, }, ], switch: Switch0, uplink_port_fec: None, uplink_port_speed: Speed40G, }, ], rack_subnet: Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56, }, } file = sled-agent/src/rack_setup/service.rs:817
22182024-05-15T07:12:23.307ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/acb2a796-8e9f-4811-b601-726a12d79b0b): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:619 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
22192024-05-15T07:12:23.314ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/89a4c48a-3dff-478b-98cb-9f441a913804/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:879
22202024-05-15T07:12:23.464ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/89a4c48a-3dff-478b-98cb-9f441a913804/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:879
22212024-05-15T07:12:23.550ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/89a4c48a-3dff-478b-98cb-9f441a913804/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:879
22222024-05-15T07:12:23.627ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/89a4c48a-3dff-478b-98cb-9f441a913804/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:879
22232024-05-15T07:12:23.807ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/89a4c48a-3dff-478b-98cb-9f441a913804/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:879
22242024-05-15T07:12:23.916ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/89a4c48a-3dff-478b-98cb-9f441a913804/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:879
22252024-05-15T07:12:24.192ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/89a4c48a-3dff-478b-98cb-9f441a913804/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:879
22262024-05-15T07:12:24.496ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/89a4c48a-3dff-478b-98cb-9f441a913804/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:879
22272024-05-15T07:12:24.969ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/89a4c48a-3dff-478b-98cb-9f441a913804/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:879
22282024-05-15T07:12:25.075ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/acb2a796-8e9f-4811-b601-726a12d79b0b): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:619 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
22292024-05-15T07:12:25.188ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/89a4c48a-3dff-478b-98cb-9f441a913804/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:879
22302024-05-15T07:12:25.560ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/89a4c48a-3dff-478b-98cb-9f441a913804/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:879
22312024-05-15T07:12:25.942ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/89a4c48a-3dff-478b-98cb-9f441a913804/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:879
22322024-05-15T07:12:26.991ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/89a4c48a-3dff-478b-98cb-9f441a913804/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:879
22332024-05-15T07:12:27.075ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/acb2a796-8e9f-4811-b601-726a12d79b0b): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:619 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
22342024-05-15T07:12:28.106ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/89a4c48a-3dff-478b-98cb-9f441a913804/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:879
22352024-05-15T07:12:29.070ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/acb2a796-8e9f-4811-b601-726a12d79b0b): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:619 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
22362024-05-15T07:12:29.095ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/acb2a796-8e9f-4811-b601-726a12d79b0b): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:169 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
22372024-05-15T07:12:29.438ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/89a4c48a-3dff-478b-98cb-9f441a913804/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:879
22382024-05-15T07:12:30.098ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/acb2a796-8e9f-4811-b601-726a12d79b0b): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:169 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
22392024-05-15T07:12:30.878ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/89a4c48a-3dff-478b-98cb-9f441a913804/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:879
22402024-05-15T07:12:31.066ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/acb2a796-8e9f-4811-b601-726a12d79b0b): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:619 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
22412024-05-15T07:12:31.101ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/acb2a796-8e9f-4811-b601-726a12d79b0b): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:169 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
22422024-05-15T07:12:32.104ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/acb2a796-8e9f-4811-b601-726a12d79b0b): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:169 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
22432024-05-15T07:12:32.894ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 403 Forbidden; headers: {"content-type": "application/json", "x-request-id": "3d298a7a-a40d-470b-b20b-940db826d743", "content-length": "113", "date": "Wed, 15 May 2024 07:12:32 GMT"}; value: Error { error_code: Some("Forbidden"), message: "Forbidden", request_id: "3d298a7a-a40d-470b-b20b-940db826d743" } file = sled-agent/src/rack_setup/service.rs:879
22442024-05-15T07:12:33.526ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:59637
22452024-05-15T07:12:33.527ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 237 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:59637 req_id = 5c42073e-5377-4592-9def-5e72c6986bd7 response_code = 200 uri = /baseboard
22462024-05-15T07:12:34.987ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "b79cbd22-932d-4a9d-a4c4-c77c63bd79f5", "content-length": "173", "date": "Wed, 15 May 2024 07:12:34 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"acb2a796-8e9f-4811-b601-726a12d79b0b\"", request_id: "b79cbd22-932d-4a9d-a4c4-c77c63bd79f5" } file = sled-agent/src/rack_setup/service.rs:879
22472024-05-15T07:12:35.131ZINFOSledAgent (NexusNotifierTask): Successfully put SledAgentInfo to nexus file = sled-agent/src/nexus.rs:592 new_generation = 1 old_generation = 1 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
22482024-05-15T07:12:35.257ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:37863
22492024-05-15T07:12:35.259ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
22502024-05-15T07:12:35.352ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 94159 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:37863 req_id = 8bb9e1e4-828b-480f-8dde-3edab2b98673 response_code = 200 uri = /inventory
22512024-05-15T07:12:35.354ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
22522024-05-15T07:12:35.354ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
22532024-05-15T07:12:35.356ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
22542024-05-15T07:12:35.356ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 4215 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:37863 req_id = 205076cb-ce30-4ac6-8e07-166edaceb5a8 response_code = 200 uri = /omicron-zones
22552024-05-15T07:12:38.779ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61404
22562024-05-15T07:12:38.782ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:525 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 2e30c7c4-8de6-4cbd-8a3d-052bb77d3985, kind: Service { id: e5538333-fcad-480c-9181-aa7e94aaf834 }, name: Name("external-dns-e5538333-fcad-480c-9181-aa7e94aaf834"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 204, 142])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: fe506ce4-c84e-4ef5-9ed5-5c96a847dd43, kind: Service { id: 12908044-fccf-4c46-8e7c-c2a0be1bfb25 }, name: Name("external-dns-12908044-fccf-4c46-8e7c-c2a0be1bfb25"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 70])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 38eee15b-cb30-4e2c-b5c3-f5511aac6743, kind: Service { id: 99861af6-438d-4bef-ba93-632645b46d2f }, name: Name("nexus-99861af6-438d-4bef-ba93-632645b46d2f"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 222, 14])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 88dead63-2254-4036-85c5-6efbd3602a0d, kind: Service { id: feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3 }, name: Name("nexus-feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 207, 1])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 9b63bcc5-cf23-47dc-87dc-38f5ac4bcfd6, kind: Service { id: 9cebcc66-fa56-4e34-af63-2c7ac52ff959 }, name: Name("nexus-9cebcc66-fa56-4e34-af63-2c7ac52ff959"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 234, 140])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
22572024-05-15T07:12:38.782ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
22582024-05-15T07:12:38.782ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
22592024-05-15T07:12:38.782ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
22602024-05-15T07:12:38.782ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
22612024-05-15T07:12:38.782ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte0" rules = []
22622024-05-15T07:12:38.782ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
22632024-05-15T07:12:38.782ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 3180 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:61404 req_id = 02e530e1-39fc-4701-a212-3b271b8e3b73 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
22642024-05-15T07:12:38.785ZINFOSledAgent (RSS): Handoff to Nexus is complete file = sled-agent/src/rack_setup/service.rs:889
22652024-05-15T07:12:38.934ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57963
22662024-05-15T07:12:38.934ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
22672024-05-15T07:12:39.041ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 107667 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:57963 req_id = 9d8dc2ca-ff05-41df-a8e2-ba3fe70befbb response_code = 200 uri = /inventory
22682024-05-15T07:12:39.042ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:43216
22692024-05-15T07:12:39.042ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
22702024-05-15T07:12:39.042ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/config/omicron-zones.json file = common/src/ledger.rs:177
22712024-05-15T07:12:39.042ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:525 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 2e30c7c4-8de6-4cbd-8a3d-052bb77d3985, kind: Service { id: e5538333-fcad-480c-9181-aa7e94aaf834 }, name: Name("external-dns-e5538333-fcad-480c-9181-aa7e94aaf834"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 204, 142])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: fe506ce4-c84e-4ef5-9ed5-5c96a847dd43, kind: Service { id: 12908044-fccf-4c46-8e7c-c2a0be1bfb25 }, name: Name("external-dns-12908044-fccf-4c46-8e7c-c2a0be1bfb25"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 70])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 38eee15b-cb30-4e2c-b5c3-f5511aac6743, kind: Service { id: 99861af6-438d-4bef-ba93-632645b46d2f }, name: Name("nexus-99861af6-438d-4bef-ba93-632645b46d2f"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 222, 14])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 88dead63-2254-4036-85c5-6efbd3602a0d, kind: Service { id: feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3 }, name: Name("nexus-feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 207, 1])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 9b63bcc5-cf23-47dc-87dc-38f5ac4bcfd6, kind: Service { id: 9cebcc66-fa56-4e34-af63-2c7ac52ff959 }, name: Name("nexus-9cebcc66-fa56-4e34-af63-2c7ac52ff959"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 234, 140])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
22722024-05-15T07:12:39.042ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
22732024-05-15T07:12:39.042ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
22742024-05-15T07:12:39.042ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
22752024-05-15T07:12:39.042ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
22762024-05-15T07:12:39.042ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte0" rules = []
22772024-05-15T07:12:39.042ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
22782024-05-15T07:12:39.042ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 592 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:43216 req_id = 9ad20360-890f-488d-ae51-0f889648f461 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
22792024-05-15T07:12:39.042ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/config/omicron-zones.json file = common/src/ledger.rs:177
22802024-05-15T07:12:39.043ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 1009 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:57963 req_id = 1a0a8a9d-6087-4119-9f1a-9929afb0490f response_code = 200 uri = /omicron-zones
22812024-05-15T07:12:39.062ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:36449
22822024-05-15T07:12:39.063ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:525 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 2e30c7c4-8de6-4cbd-8a3d-052bb77d3985, kind: Service { id: e5538333-fcad-480c-9181-aa7e94aaf834 }, name: Name("external-dns-e5538333-fcad-480c-9181-aa7e94aaf834"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 204, 142])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: fe506ce4-c84e-4ef5-9ed5-5c96a847dd43, kind: Service { id: 12908044-fccf-4c46-8e7c-c2a0be1bfb25 }, name: Name("external-dns-12908044-fccf-4c46-8e7c-c2a0be1bfb25"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 70])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 38eee15b-cb30-4e2c-b5c3-f5511aac6743, kind: Service { id: 99861af6-438d-4bef-ba93-632645b46d2f }, name: Name("nexus-99861af6-438d-4bef-ba93-632645b46d2f"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 222, 14])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 88dead63-2254-4036-85c5-6efbd3602a0d, kind: Service { id: feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3 }, name: Name("nexus-feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 207, 1])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 9b63bcc5-cf23-47dc-87dc-38f5ac4bcfd6, kind: Service { id: 9cebcc66-fa56-4e34-af63-2c7ac52ff959 }, name: Name("nexus-9cebcc66-fa56-4e34-af63-2c7ac52ff959"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 234, 140])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
22832024-05-15T07:12:39.063ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
22842024-05-15T07:12:39.063ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
22852024-05-15T07:12:39.063ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
22862024-05-15T07:12:39.063ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
22872024-05-15T07:12:39.063ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte0" rules = []
22882024-05-15T07:12:39.063ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
22892024-05-15T07:12:39.063ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 899 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:36449 req_id = ba17ed3d-9f39-445c-830a-acdde3c7bbf1 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
22902024-05-15T07:12:39.119ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:35769
22912024-05-15T07:12:39.119ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:525 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 2e30c7c4-8de6-4cbd-8a3d-052bb77d3985, kind: Service { id: e5538333-fcad-480c-9181-aa7e94aaf834 }, name: Name("external-dns-e5538333-fcad-480c-9181-aa7e94aaf834"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 204, 142])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: fe506ce4-c84e-4ef5-9ed5-5c96a847dd43, kind: Service { id: 12908044-fccf-4c46-8e7c-c2a0be1bfb25 }, name: Name("external-dns-12908044-fccf-4c46-8e7c-c2a0be1bfb25"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 70])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 38eee15b-cb30-4e2c-b5c3-f5511aac6743, kind: Service { id: 99861af6-438d-4bef-ba93-632645b46d2f }, name: Name("nexus-99861af6-438d-4bef-ba93-632645b46d2f"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 222, 14])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 88dead63-2254-4036-85c5-6efbd3602a0d, kind: Service { id: feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3 }, name: Name("nexus-feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 207, 1])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 9b63bcc5-cf23-47dc-87dc-38f5ac4bcfd6, kind: Service { id: 9cebcc66-fa56-4e34-af63-2c7ac52ff959 }, name: Name("nexus-9cebcc66-fa56-4e34-af63-2c7ac52ff959"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 234, 140])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
22922024-05-15T07:12:39.119ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
22932024-05-15T07:12:39.119ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
22942024-05-15T07:12:39.119ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
22952024-05-15T07:12:39.119ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
22962024-05-15T07:12:39.119ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte0" rules = []
22972024-05-15T07:12:39.119ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
22982024-05-15T07:12:39.119ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 317 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:35769 req_id = 3d7e6a9a-1e6a-41a5-a4f0-fb0ddb8a65c7 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
22992024-05-15T07:13:02.948ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52531
23002024-05-15T07:13:03.196ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 248208 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:52531 req_id = 28bc9a69-71bd-4fc2-8bac-5108a2ff243b response_code = 204 uri = /switch-ports
23012024-05-15T07:13:03.197ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:49544
23022024-05-15T07:13:03.197ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:43840
23032024-05-15T07:13:03.207ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 10586 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:52531 req_id = 9f4826e5-c9ca-4732-ae85-483f78b1ea79 response_code = 200 uri = /network-bootstore-config
23042024-05-15T07:13:03.264ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 67436 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:43840 req_id = e0aee942-eb3e-439a-880e-b79223f15578 response_code = 204 uri = /switch-ports
23052024-05-15T07:13:03.268ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 667 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:43840 req_id = 8299ffd6-6b63-4d3c-9639-696ca9d6d066 response_code = 200 uri = /network-bootstore-config
23062024-05-15T07:13:03.302ZINFOSledAgent (bootstore): Attempting to update network config with generation: 2, current_generation: 1 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-sock-i86pc
23072024-05-15T07:13:03.310ZINFOSledAgent (bootstore): Writing ledger to /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
23082024-05-15T07:13:03.315ZINFOSledAgent (bootstore): Writing ledger to /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
23092024-05-15T07:13:03.316ZINFOSledAgent (bootstore): Broadcasting network config with generation 2 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-sock-i86pc
23102024-05-15T07:13:03.316ZINFOSledAgent (bootstore): Attempting to update network config with generation: 3, current_generation: 2 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-sock-i86pc
23112024-05-15T07:13:03.316ZINFOSledAgent (bootstore): Writing ledger to /pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
23122024-05-15T07:13:03.316ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 26688 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:52531 req_id = 7b200b6c-15f2-4e13-b9f5-9b9bf79a627d response_code = 204 uri = /network-bootstore-config
23132024-05-15T07:13:03.316ZINFOSledAgent (bootstore): Writing ledger to /pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
23142024-05-15T07:13:03.316ZINFOSledAgent (bootstore): Broadcasting network config with generation 3 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-sock-i86pc
23152024-05-15T07:13:03.316ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 13480 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:43840 req_id = f437cb59-2aa6-433a-9663-0f1b24219639 response_code = 204 uri = /network-bootstore-config
23162024-05-15T07:13:03.323ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 126368 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:49544 req_id = ef35cae4-c56a-49ee-92bf-a967c27388a5 response_code = 204 uri = /switch-ports
23172024-05-15T07:13:03.323ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:49544 req_id = 05a1649b-612e-4a55-b5a8-8f7d34ece348 response_code = 200 uri = /network-bootstore-config
23182024-05-15T07:13:03.560ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:45410
23192024-05-15T07:13:03.560ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 36 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:45410 req_id = 233b3317-ec58-479a-869f-a47c4c40cae9 response_code = 200 uri = /baseboard
23202024-05-15T07:13:20.838ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
23212024-05-15T07:13:33.135ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:44762
23222024-05-15T07:13:33.185ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 50064 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:44762 req_id = 398cfca2-28fc-4e98-a923-0c771c19a7cb response_code = 204 uri = /switch-ports
23232024-05-15T07:13:33.185ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:35727
23242024-05-15T07:13:33.189ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 3185 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:44762 req_id = f865349d-bf7a-4c34-ab14-2300ce7b3dce response_code = 200 uri = /network-bootstore-config
23252024-05-15T07:13:33.202ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:55652
23262024-05-15T07:13:33.235ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 49954 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:35727 req_id = 8ca03e7a-ac82-483d-a49b-ed393364ed78 response_code = 204 uri = /switch-ports
23272024-05-15T07:13:33.239ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 986 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:35727 req_id = 00fc9f5f-9aa9-45e7-85c5-8fe7afc30c4f response_code = 200 uri = /network-bootstore-config
23282024-05-15T07:13:33.282ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 79262 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:55652 req_id = 7b7de0ae-6d0c-4501-a248-62740839f63f response_code = 204 uri = /switch-ports
23292024-05-15T07:13:33.282ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 127 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:55652 req_id = 4a221d60-f3ec-4e3a-9fd9-8a6de2c39908 response_code = 200 uri = /network-bootstore-config
23302024-05-15T07:13:33.594ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:34286
23312024-05-15T07:13:33.594ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:34286 req_id = b4981772-3d6f-49c7-b1d8-9bf0a075ec30 response_code = 200 uri = /baseboard
23322024-05-15T07:14:02.909ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:60996
23332024-05-15T07:14:02.959ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 49424 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:60996 req_id = beddff37-a382-4d61-904f-aba722256fe0 response_code = 204 uri = /switch-ports
23342024-05-15T07:14:02.959ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 210 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:60996 req_id = 3f578d53-8574-46fa-9aa8-d03c564af5ea response_code = 200 uri = /network-bootstore-config
23352024-05-15T07:14:02.968ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:35320
23362024-05-15T07:14:03.015ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 47309 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:35320 req_id = da7002e7-726e-4a6f-9c72-d82f4f76d353 response_code = 204 uri = /switch-ports
23372024-05-15T07:14:03.015ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:64339
23382024-05-15T07:14:03.019ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 3709 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:35320 req_id = b614e5d8-1a22-48a1-8c33-713d98f1ffed response_code = 200 uri = /network-bootstore-config
23392024-05-15T07:14:03.062ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 46405 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:64339 req_id = daff3b85-32b6-4627-ac8b-a61a917fee18 response_code = 204 uri = /switch-ports
23402024-05-15T07:14:03.062ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 167 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:64339 req_id = b1045de8-73c2-48e8-947a-5e584d0505b4 response_code = 200 uri = /network-bootstore-config
23412024-05-15T07:14:03.628ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:44913
23422024-05-15T07:14:03.629ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 748 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:44913 req_id = 04b63b70-1d66-42ed-8a18-0b6c424f9eed response_code = 200 uri = /baseboard
23432024-05-15T07:14:10.263ZINFOSledAgent (dropshot): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:55688 remote_addr = [fd00:1122:3344:101::d]:65149 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b
23442024-05-15T07:14:10.271ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 6994 local_addr = [fd00:1122:3344:101::1]:55688 method = GET remote_addr = [fd00:1122:3344:101::d]:65149 req_id = 05e61014-d288-48f3-abc1-40686d810572 response_code = 200 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b uri = //acb2a796-8e9f-4811-b601-726a12d79b0b
23452024-05-15T07:14:33.179ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63438
23462024-05-15T07:14:33.228ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 49360 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:63438 req_id = b2332f70-30d8-4049-9ff6-a90999c57f69 response_code = 204 uri = /switch-ports
23472024-05-15T07:14:33.228ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 100 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:63438 req_id = b6a6d308-ebad-40ab-b786-febd7cd991e2 response_code = 200 uri = /network-bootstore-config
23482024-05-15T07:14:33.310ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:52997
23492024-05-15T07:14:33.359ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 49443 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:52997 req_id = 1028b17c-24b6-4f71-8afb-957a02735929 response_code = 204 uri = /switch-ports
23502024-05-15T07:14:33.359ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:35614
23512024-05-15T07:14:33.360ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 213 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:52997 req_id = 5efed80f-370e-4a01-b84c-10560607c20a response_code = 200 uri = /network-bootstore-config
23522024-05-15T07:14:33.407ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 47495 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:35614 req_id = c132f461-b434-4b96-b0ca-c93b8869a117 response_code = 204 uri = /switch-ports
23532024-05-15T07:14:33.407ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 147 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:35614 req_id = 59e9806c-713d-4ed5-b8b7-57552c509996 response_code = 200 uri = /network-bootstore-config
23542024-05-15T07:14:33.660ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:48814
23552024-05-15T07:14:33.661ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 89 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:48814 req_id = 1fc36acd-4f1d-4592-93c1-03aa46db3acb response_code = 200 uri = /baseboard
23562024-05-15T07:14:40.262ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 1233 local_addr = [fd00:1122:3344:101::1]:55688 method = GET remote_addr = [fd00:1122:3344:101::d]:65149 req_id = 6c4a0b17-7b19-4008-8484-8da5257a493b response_code = 200 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b uri = //acb2a796-8e9f-4811-b601-726a12d79b0b
23572024-05-15T07:15:02.981ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:60589
23582024-05-15T07:15:03.041ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 59481 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:60589 req_id = 36449bb0-3ac6-4f3e-aa44-54a7b930ca8e response_code = 204 uri = /switch-ports
23592024-05-15T07:15:03.041ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:35240
23602024-05-15T07:15:03.089ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 48105 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:35240 req_id = 80c893df-17ee-488d-bd0e-5dfc8ff37d63 response_code = 204 uri = /switch-ports
23612024-05-15T07:15:03.089ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 107 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:60589 req_id = 0f9b5950-51f3-4004-a26a-75fe609dbf60 response_code = 200 uri = /network-bootstore-config
23622024-05-15T07:15:03.090ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 791 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:35240 req_id = 6d67ada2-1e64-45ad-9edd-9f5da103713d response_code = 200 uri = /network-bootstore-config
23632024-05-15T07:15:03.107ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62753
23642024-05-15T07:15:03.156ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 48741 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:62753 req_id = 964f7f9d-0b46-4039-a788-9acb3ffd61c5 response_code = 204 uri = /switch-ports
23652024-05-15T07:15:03.156ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 108 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:62753 req_id = 7f82a5fe-dd7a-4401-8325-2324b157352e response_code = 200 uri = /network-bootstore-config
23662024-05-15T07:15:03.694ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:51655
23672024-05-15T07:15:03.694ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 78 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:51655 req_id = 12776a45-b514-45db-a893-f7b27ff570ef response_code = 200 uri = /baseboard
23682024-05-15T07:15:10.262ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 775 local_addr = [fd00:1122:3344:101::1]:55688 method = GET remote_addr = [fd00:1122:3344:101::d]:65149 req_id = dec04add-b9ac-4431-ab87-3dd1cfb7340e response_code = 200 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b uri = //acb2a796-8e9f-4811-b601-726a12d79b0b
23692024-05-15T07:15:33.101ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:54570
23702024-05-15T07:15:33.152ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 51104 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:54570 req_id = 3bc4c5cb-489e-4fb5-9c7a-c6ab1063fc22 response_code = 204 uri = /switch-ports
23712024-05-15T07:15:33.153ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33472
23722024-05-15T07:15:33.156ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 3361 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:54570 req_id = aa61d6bc-2b3e-4d02-9842-4a2a94bf2b25 response_code = 200 uri = /network-bootstore-config
23732024-05-15T07:15:33.199ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 46362 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:33472 req_id = 8a27fe0f-1913-4d4a-b951-c1a1e59e9544 response_code = 204 uri = /switch-ports
23742024-05-15T07:15:33.200ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 1108 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:33472 req_id = d3e9e866-417a-41ae-9ace-e013b1e270b5 response_code = 200 uri = /network-bootstore-config
23752024-05-15T07:15:33.243ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58717
23762024-05-15T07:15:33.290ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 46535 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:58717 req_id = c99d36af-89c0-46b0-8dcc-ec616b368cfa response_code = 204 uri = /switch-ports
23772024-05-15T07:15:33.290ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58717 req_id = 093c0b29-54eb-4157-954a-72e6f664193c response_code = 200 uri = /network-bootstore-config
23782024-05-15T07:15:33.728ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:37563
23792024-05-15T07:15:33.728ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 77 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:37563 req_id = 8ad307c4-0365-4827-b8af-9311d6d0b03b response_code = 200 uri = /baseboard
23802024-05-15T07:15:40.261ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 591 local_addr = [fd00:1122:3344:101::1]:55688 method = GET remote_addr = [fd00:1122:3344:101::d]:65149 req_id = 399eac94-8b08-466d-9316-7c24e014a305 response_code = 200 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b uri = //acb2a796-8e9f-4811-b601-726a12d79b0b
23812024-05-15T07:16:02.908ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:34134
23822024-05-15T07:16:02.958ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 49974 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:34134 req_id = b88c276e-68d2-4285-9b07-3f559b8d14dc response_code = 204 uri = /switch-ports
23832024-05-15T07:16:02.958ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:34134 req_id = da56b1a8-3983-4363-b7c1-b59de8d17585 response_code = 200 uri = /network-bootstore-config
23842024-05-15T07:16:02.984ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42671
23852024-05-15T07:16:03.031ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 46892 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:42671 req_id = dcd2c5fb-0918-4af0-88b2-860a9228f29c response_code = 204 uri = /switch-ports
23862024-05-15T07:16:03.031ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 103 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:42671 req_id = b27630f0-7534-4c06-b11d-c6ead4a898c1 response_code = 200 uri = /network-bootstore-config
23872024-05-15T07:16:03.075ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:56470
23882024-05-15T07:16:03.122ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 46565 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:56470 req_id = ea07257b-e0ce-4abe-a469-6c9d36ca3665 response_code = 204 uri = /switch-ports
23892024-05-15T07:16:03.122ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:56470 req_id = 95ca3787-0fc2-495c-a824-675f651e3d4e response_code = 200 uri = /network-bootstore-config
23902024-05-15T07:16:03.769ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:61240
23912024-05-15T07:16:03.769ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 99 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:61240 req_id = 6ef44f29-34d2-4950-80f5-2b577cc1aa84 response_code = 200 uri = /baseboard
23922024-05-15T07:16:10.274ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 884 local_addr = [fd00:1122:3344:101::1]:55688 method = GET remote_addr = [fd00:1122:3344:101::d]:65149 req_id = 03d350f2-1166-4f34-a506-659ce5186e17 response_code = 200 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b uri = //acb2a796-8e9f-4811-b601-726a12d79b0b
23932024-05-15T07:16:33.118ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:59864
23942024-05-15T07:16:33.168ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 49414 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:59864 req_id = ef700789-750d-4890-909d-faec180e2044 response_code = 204 uri = /switch-ports
23952024-05-15T07:16:33.168ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:59864 req_id = 75d82a7e-a12b-4d4a-b95e-11dd8637b425 response_code = 200 uri = /network-bootstore-config
23962024-05-15T07:16:33.175ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:37882
23972024-05-15T07:16:33.225ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 49116 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:37882 req_id = ceeff066-b412-41de-9722-a6d6bc38e5b2 response_code = 204 uri = /switch-ports
23982024-05-15T07:16:33.225ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44747
23992024-05-15T07:16:33.228ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 3334 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:37882 req_id = 3b130e9e-97e5-4cbb-8762-514d34343a0c response_code = 200 uri = /network-bootstore-config
24002024-05-15T07:16:33.271ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 46224 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:44747 req_id = 8f77a1dd-6d51-4a50-b3ad-73f2001f9d06 response_code = 204 uri = /switch-ports
24012024-05-15T07:16:33.271ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 137 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44747 req_id = 82ec753d-08ba-49a6-8f2d-c5930aef8540 response_code = 200 uri = /network-bootstore-config
24022024-05-15T07:16:33.801ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:47426
24032024-05-15T07:16:33.801ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 69 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:47426 req_id = e26513a4-0a72-4b78-9897-bcfd91e5fb3f response_code = 200 uri = /baseboard
24042024-05-15T07:16:40.310ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 735 local_addr = [fd00:1122:3344:101::1]:55688 method = GET remote_addr = [fd00:1122:3344:101::d]:65149 req_id = 33088b5c-34c2-4641-b528-cb827b6fe05f response_code = 200 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b uri = //acb2a796-8e9f-4811-b601-726a12d79b0b
24052024-05-15T07:17:02.915ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36935
24062024-05-15T07:17:02.966ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 50315 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:36935 req_id = 5bca168d-11fb-4e93-bc06-e656ea964358 response_code = 204 uri = /switch-ports
24072024-05-15T07:17:02.966ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:56807
24082024-05-15T07:17:02.970ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 3963 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:36935 req_id = 994624f3-5f7e-4674-bdb7-94bd04721d04 response_code = 200 uri = /network-bootstore-config
24092024-05-15T07:17:03.015ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 49240 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:56807 req_id = addcbcc2-6626-4519-a575-0103b6112dbf response_code = 204 uri = /switch-ports
24102024-05-15T07:17:03.015ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 133 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:56807 req_id = 156b7690-1207-4547-b988-2b559a85a53a response_code = 200 uri = /network-bootstore-config
24112024-05-15T07:17:03.089ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:33083
24122024-05-15T07:17:03.134ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 44669 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:33083 req_id = c8eaabf7-6951-4f6b-b89b-1083afd1af9f response_code = 204 uri = /switch-ports
24132024-05-15T07:17:03.134ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 106 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:33083 req_id = 83221fb5-99b6-4fc0-ad1f-75385bd81229 response_code = 200 uri = /network-bootstore-config
24142024-05-15T07:17:03.832ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:44995
24152024-05-15T07:17:03.832ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 89 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:44995 req_id = 894d0526-cf1f-4676-8933-ad4c04fffc57 response_code = 200 uri = /baseboard
24162024-05-15T07:17:10.310ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 620 local_addr = [fd00:1122:3344:101::1]:55688 method = GET remote_addr = [fd00:1122:3344:101::d]:65149 req_id = a97ae30f-2b33-4f80-be98-ef985a0304bc response_code = 200 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b uri = //acb2a796-8e9f-4811-b601-726a12d79b0b
24172024-05-15T07:17:33.045ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:45132
24182024-05-15T07:17:33.045ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:525 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 2e30c7c4-8de6-4cbd-8a3d-052bb77d3985, kind: Service { id: e5538333-fcad-480c-9181-aa7e94aaf834 }, name: Name("external-dns-e5538333-fcad-480c-9181-aa7e94aaf834"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 204, 142])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: fe506ce4-c84e-4ef5-9ed5-5c96a847dd43, kind: Service { id: 12908044-fccf-4c46-8e7c-c2a0be1bfb25 }, name: Name("external-dns-12908044-fccf-4c46-8e7c-c2a0be1bfb25"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 70])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 38eee15b-cb30-4e2c-b5c3-f5511aac6743, kind: Service { id: 99861af6-438d-4bef-ba93-632645b46d2f }, name: Name("nexus-99861af6-438d-4bef-ba93-632645b46d2f"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 222, 14])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 88dead63-2254-4036-85c5-6efbd3602a0d, kind: Service { id: feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3 }, name: Name("nexus-feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 207, 1])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 9b63bcc5-cf23-47dc-87dc-38f5ac4bcfd6, kind: Service { id: 9cebcc66-fa56-4e34-af63-2c7ac52ff959 }, name: Name("nexus-9cebcc66-fa56-4e34-af63-2c7ac52ff959"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 234, 140])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
24192024-05-15T07:17:33.045ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
24202024-05-15T07:17:33.045ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
24212024-05-15T07:17:33.045ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
24222024-05-15T07:17:33.046ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
24232024-05-15T07:17:33.046ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte0" rules = []
24242024-05-15T07:17:33.046ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
24252024-05-15T07:17:33.046ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 377 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:45132 req_id = c116ff48-a53b-4574-80ca-116b0fb617e1 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
24262024-05-15T07:17:33.101ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:60191
24272024-05-15T07:17:33.149ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 48081 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:60191 req_id = d5e06d5a-dbc6-4193-be79-ddecfb4b0aef response_code = 204 uri = /switch-ports
24282024-05-15T07:17:33.150ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:56776
24292024-05-15T07:17:33.150ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 187 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:60191 req_id = 0d55b02d-f9d3-4c1e-91c3-2ccd1f44e1f1 response_code = 200 uri = /network-bootstore-config
24302024-05-15T07:17:33.150ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:525 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 2e30c7c4-8de6-4cbd-8a3d-052bb77d3985, kind: Service { id: e5538333-fcad-480c-9181-aa7e94aaf834 }, name: Name("external-dns-e5538333-fcad-480c-9181-aa7e94aaf834"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 204, 142])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: fe506ce4-c84e-4ef5-9ed5-5c96a847dd43, kind: Service { id: 12908044-fccf-4c46-8e7c-c2a0be1bfb25 }, name: Name("external-dns-12908044-fccf-4c46-8e7c-c2a0be1bfb25"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 70])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 38eee15b-cb30-4e2c-b5c3-f5511aac6743, kind: Service { id: 99861af6-438d-4bef-ba93-632645b46d2f }, name: Name("nexus-99861af6-438d-4bef-ba93-632645b46d2f"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 222, 14])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 88dead63-2254-4036-85c5-6efbd3602a0d, kind: Service { id: feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3 }, name: Name("nexus-feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 207, 1])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 9b63bcc5-cf23-47dc-87dc-38f5ac4bcfd6, kind: Service { id: 9cebcc66-fa56-4e34-af63-2c7ac52ff959 }, name: Name("nexus-9cebcc66-fa56-4e34-af63-2c7ac52ff959"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 234, 140])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
24312024-05-15T07:17:33.150ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
24322024-05-15T07:17:33.150ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
24332024-05-15T07:17:33.150ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
24342024-05-15T07:17:33.150ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
24352024-05-15T07:17:33.150ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte0" rules = []
24362024-05-15T07:17:33.150ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
24372024-05-15T07:17:33.150ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 496 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:56776 req_id = a88ed11f-061b-4ebc-aac1-8ebcbb4ecca3 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
24382024-05-15T07:17:33.169ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34152
24392024-05-15T07:17:33.169ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:525 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 2e30c7c4-8de6-4cbd-8a3d-052bb77d3985, kind: Service { id: e5538333-fcad-480c-9181-aa7e94aaf834 }, name: Name("external-dns-e5538333-fcad-480c-9181-aa7e94aaf834"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 204, 142])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: fe506ce4-c84e-4ef5-9ed5-5c96a847dd43, kind: Service { id: 12908044-fccf-4c46-8e7c-c2a0be1bfb25 }, name: Name("external-dns-12908044-fccf-4c46-8e7c-c2a0be1bfb25"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 70])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 38eee15b-cb30-4e2c-b5c3-f5511aac6743, kind: Service { id: 99861af6-438d-4bef-ba93-632645b46d2f }, name: Name("nexus-99861af6-438d-4bef-ba93-632645b46d2f"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 222, 14])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 88dead63-2254-4036-85c5-6efbd3602a0d, kind: Service { id: feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3 }, name: Name("nexus-feeb24cb-4b3b-4c4b-b3ef-d86f60b83ea3"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 207, 1])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 9b63bcc5-cf23-47dc-87dc-38f5ac4bcfd6, kind: Service { id: 9cebcc66-fa56-4e34-af63-2c7ac52ff959 }, name: Name("nexus-9cebcc66-fa56-4e34-af63-2c7ac52ff959"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 234, 140])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
24402024-05-15T07:17:33.169ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
24412024-05-15T07:17:33.170ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
24422024-05-15T07:17:33.170ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
24432024-05-15T07:17:33.170ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
24442024-05-15T07:17:33.170ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte0" rules = []
24452024-05-15T07:17:33.170ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
24462024-05-15T07:17:33.170ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 406 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:34152 req_id = 59663957-072b-4778-9039-c175fd8162c0 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
24472024-05-15T07:17:33.229ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58296
24482024-05-15T07:17:33.279ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 49289 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:58296 req_id = 27187d12-4698-475e-8a1e-3b035f150a53 response_code = 204 uri = /switch-ports
24492024-05-15T07:17:33.279ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 117 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58296 req_id = f8acf92b-29c1-473e-8659-7ccbbc752d1a response_code = 200 uri = /network-bootstore-config
24502024-05-15T07:17:33.294ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33458
24512024-05-15T07:17:33.339ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 45250 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:33458 req_id = 6df35e5d-75ff-4de8-84f3-0f54314c672e response_code = 204 uri = /switch-ports
24522024-05-15T07:17:33.339ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 203 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:33458 req_id = 1d4481a5-ac6c-449c-aacf-2cb7263fb11d response_code = 200 uri = /network-bootstore-config
24532024-05-15T07:17:33.865ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:64069
24542024-05-15T07:17:33.865ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 99 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:64069 req_id = bbca4da6-1609-44b9-8c76-a5bbc05b22a4 response_code = 200 uri = /baseboard
24552024-05-15T07:17:40.310ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 788 local_addr = [fd00:1122:3344:101::1]:55688 method = GET remote_addr = [fd00:1122:3344:101::d]:65149 req_id = 5f1cf87e-c336-482a-8e99-c26151e53f60 response_code = 200 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b uri = //acb2a796-8e9f-4811-b601-726a12d79b0b
24562024-05-15T07:18:02.905ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:51220
24572024-05-15T07:18:02.955ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 49848 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:51220 req_id = bcc55ac7-fec5-496b-96aa-81fb815089a9 response_code = 204 uri = /switch-ports
24582024-05-15T07:18:02.955ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:55699
24592024-05-15T07:18:02.955ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:38710
24602024-05-15T07:18:02.961ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 5695 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:51220 req_id = 31d894db-2652-408b-97cd-ca3f2eec2ba4 response_code = 200 uri = /network-bootstore-config
24612024-05-15T07:18:03.004ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 48873 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:55699 req_id = 4f7f0d85-b9b5-49b9-85b8-fad07cc24570 response_code = 204 uri = /switch-ports
24622024-05-15T07:18:03.009ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 1341 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:55699 req_id = a3e8aab2-6689-4066-a1de-434396b22abe response_code = 200 uri = /network-bootstore-config
24632024-05-15T07:18:03.051ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 94970 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:38710 req_id = 1ef7ee96-d501-4cd0-8955-ab386abbeddd response_code = 204 uri = /switch-ports
24642024-05-15T07:18:03.051ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 112 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:38710 req_id = 729c1575-4c82-4072-949f-a5459e45c444 response_code = 200 uri = /network-bootstore-config
24652024-05-15T07:18:03.899ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:53753
24662024-05-15T07:18:03.899ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 62 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:53753 req_id = 3a98d705-2216-4d0c-95e9-839e0d56bd71 response_code = 200 uri = /baseboard
24672024-05-15T07:18:07.237ZINFOSledAgent (auto-cleanup-task): running automatic periodic zone bundle cleanup file = sled-agent/src/zone_bundle.rs:300
24682024-05-15T07:18:07.238ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
24692024-05-15T07:18:07.276ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:289 next_cleanup = Instant { tv_sec: 1851, tv_nsec: 684877306 } time_to_next_cleanup = 599.999997091s
24702024-05-15T07:18:10.309ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 456 local_addr = [fd00:1122:3344:101::1]:55688 method = GET remote_addr = [fd00:1122:3344:101::d]:65149 req_id = b92bca1d-92c7-4f12-b3c8-084def773135 response_code = 200 sled_id = acb2a796-8e9f-4811-b601-726a12d79b0b uri = //acb2a796-8e9f-4811-b601-726a12d79b0b
24712024-05-15T07:18:17.269ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:32773
24722024-05-15T07:18:17.279ZINFOSledAgent (InstanceManager): ensuring instance is registered file = sled-agent/src/instance_manager.rs:546 hardware = InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: 6a12a6a1-9861-4fe8-bb03-f96ecb313186, kind: Instance { id: 74b5d93e-2241-4319-8129-fe67108c8f49 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 241, 86, 179])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(12006808), primary: true, slot: 0 }], source_nat: SourceNatConfig { ip: 10.151.2.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.2.176), floating_ips: [], firewall_rules: [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 6a12a6a1-9861-4fe8-bb03-f96ecb313186, kind: Instance { id: 74b5d93e-2241-4319-8129-fe67108c8f49 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 241, 86, 179])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(12006808), primary: true, slot: 0 }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 6a12a6a1-9861-4fe8-bb03-f96ecb313186, kind: Instance { id: 74b5d93e-2241-4319-8129-fe67108c8f49 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 241, 86, 179])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(12006808), primary: true, slot: 0 }], filter_hosts: Some([Vpc(Vni(12006808))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 6a12a6a1-9861-4fe8-bb03-f96ecb313186, kind: Instance { id: 74b5d93e-2241-4319-8129-fe67108c8f49 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 241, 86, 179])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(12006808), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-7d72237e3cb", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: a2bdf7c7-d54c-4168-8d09-daf3c96c27c1, read_only_parent: Some(Volume { block_size: 512, id: 064e6874-7451-4713-8b9b-16ba9758e09b, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 98f450bf-ba2d-4a97-b47f-1d65c4edbb9e, key: Some("N/PJy4twLSmZ7nLWEfFW8oifHAzQ1XQV3m2X/kcGn88="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::13]:19001", "[fd00:1122:3344:101::12]:19001", "[fd00:1122:3344:101::19]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: a2bdf7c7-d54c-4168-8d09-daf3c96c27c1, key: Some("08OcIC7oIXvo3LKwtBjCBiI0Wxg52LMrRfXsu93NGs8="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::16]:19001", "[fd00:1122:3344:101::19]:19002", "[fd00:1122:3344:101::13]:19002"] } }] } }], cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAAB6SDqvWK9YAABIOq9YAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAe0g6r1ivWAAASDqvWAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiI3NGI1ZDkzZS0yMjQxLTQzMTktODEyOS1mZTY3MTA4YzhmNDkiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJTDlyZFRFMkR3WXU5WGloTWxwdDQ4ZzU3OUhjZkROMDQ0MXpKTVhIbUFUZSJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA") } instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49 instance_runtime = InstanceRuntimeState { propolis_id: Some(b926d0d2-78b1-4d16-b1ed-d24aaeaf7543), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-05-15T07:18:16.811647382Z } metadata = InstanceMetadata { silo_id: 82ce2851-ee69-4933-b9a7-f0eee8dffe12, project_id: 22d38dba-1ffb-40fc-b3f3-4ee2a548bcf6 } propolis_addr = [fd00:1122:3344:101::1:0]:12400 propolis_id = b926d0d2-78b1-4d16-b1ed-d24aaeaf7543 vmm_runtime = VmmRuntimeState { state: Starting, gen: Generation(1), time_updated: 2024-05-15T07:18:16.763717Z }
24732024-05-15T07:18:17.279ZINFOSledAgent (InstanceManager): registering new instance file = sled-agent/src/instance_manager.rs:580 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49
24742024-05-15T07:18:17.279ZINFOSledAgent (InstanceManager): initializing new Instance file = sled-agent/src/instance.rs:938 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49 propolis_id = b926d0d2-78b1-4d16-b1ed-d24aaeaf7543 state = InstanceInitialState { hardware: InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: 6a12a6a1-9861-4fe8-bb03-f96ecb313186, kind: Instance { id: 74b5d93e-2241-4319-8129-fe67108c8f49 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 241, 86, 179])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(12006808), primary: true, slot: 0 }], source_nat: SourceNatConfig { ip: 10.151.2.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.2.176), floating_ips: [], firewall_rules: [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 6a12a6a1-9861-4fe8-bb03-f96ecb313186, kind: Instance { id: 74b5d93e-2241-4319-8129-fe67108c8f49 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 241, 86, 179])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(12006808), primary: true, slot: 0 }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 6a12a6a1-9861-4fe8-bb03-f96ecb313186, kind: Instance { id: 74b5d93e-2241-4319-8129-fe67108c8f49 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 241, 86, 179])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(12006808), primary: true, slot: 0 }], filter_hosts: Some([Vpc(Vni(12006808))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 6a12a6a1-9861-4fe8-bb03-f96ecb313186, kind: Instance { id: 74b5d93e-2241-4319-8129-fe67108c8f49 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 241, 86, 179])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(12006808), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-7d72237e3cb", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: a2bdf7c7-d54c-4168-8d09-daf3c96c27c1, read_only_parent: Some(Volume { block_size: 512, id: 064e6874-7451-4713-8b9b-16ba9758e09b, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 98f450bf-ba2d-4a97-b47f-1d65c4edbb9e, key: Some("N/PJy4twLSmZ7nLWEfFW8oifHAzQ1XQV3m2X/kcGn88="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::13]:19001", "[fd00:1122:3344:101::12]:19001", "[fd00:1122:3344:101::19]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: a2bdf7c7-d54c-4168-8d09-daf3c96c27c1, key: Some("08OcIC7oIXvo3LKwtBjCBiI0Wxg52LMrRfXsu93NGs8="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::16]:19001", "[fd00:1122:3344:101::19]:19002", "[fd00:1122:3344:101::13]:19002"] } }] } }], cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAAB6SDqvWK9YAABIOq9YAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAe0g6r1ivWAAASDqvWAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiI3NGI1ZDkzZS0yMjQxLTQzMTktODEyOS1mZTY3MTA4YzhmNDkiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJTDlyZFRFMkR3WXU5WGloTWxwdDQ4ZzU3OUhjZkROMDQ0MXpKTVhIbUFUZSJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA") }, instance_runtime: InstanceRuntimeState { propolis_id: Some(b926d0d2-78b1-4d16-b1ed-d24aaeaf7543), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-05-15T07:18:16.811647382Z }, vmm_runtime: VmmRuntimeState { state: Starting, gen: Generation(1), time_updated: 2024-05-15T07:18:16.763717Z }, propolis_addr: [fd00:1122:3344:101::1:0]:12400 }
24752024-05-15T07:18:17.282ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 13118 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:32773 req_id = 1ab5419d-8210-446d-b805-9ef2d6bffd63 response_code = 200 uri = /instances/74b5d93e-2241-4319-8129-fe67108c8f49
24762024-05-15T07:18:17.397ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:55817
24772024-05-15T07:18:17.397ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 392 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:55817 req_id = a6cd5a60-9841-4bd2-95e5-e097ae602b03 response_code = 200 uri = /instances/74b5d93e-2241-4319-8129-fe67108c8f49/state
24782024-05-15T07:18:17.422ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:394 port = Port { inner: PortInner { name: "opte6", ip: 172.30.0.5, mac: MacAddr6([168, 64, 37, 241, 86, 179]), slot: 0, vni: Vni { inner: 12006808 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.0.1 }, vnic: "vopte6" } }
24792024-05-15T07:18:17.423ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
24802024-05-15T07:18:17.459ZINFOSledAgent (InstanceManager): Configuring new Omicron zone: oxz_propolis-server_b926d0d2-78b1-4d16-b1ed-d24aaeaf7543 file = illumos-utils/src/zone.rs:315 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49
24812024-05-15T07:18:17.480ZINFOSledAgent (InstanceManager): Installing Omicron zone: oxz_propolis-server_b926d0d2-78b1-4d16-b1ed-d24aaeaf7543 file = illumos-utils/src/zone.rs:354 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49
24822024-05-15T07:18:18.856ZINFOSledAgent (InstanceManager): Profile for oxz_propolis-server_b926d0d2-78b1-4d16-b1ed-d24aaeaf7543: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="system/illumos/propolis-server"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlInstance0"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::1:0"/> <propval type="astring" name="listen_port" value="12400"/> <propval type="astring" name="metric_addr" value="[fd00:1122:3344:101::c]:12221"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49
24832024-05-15T07:18:18.856ZINFOSledAgent (InstanceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49 zone = oxz_propolis-server_b926d0d2-78b1-4d16-b1ed-d24aaeaf7543
24842024-05-15T07:18:19.017ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_b926d0d2-78b1-4d16-b1ed-d24aaeaf7543") failed: Property not found. retry in 59.142966ms file = illumos-utils/src/svc.rs:36 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49 zone = oxz_propolis-server_b926d0d2-78b1-4d16-b1ed-d24aaeaf7543
24852024-05-15T07:18:19.092ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_b926d0d2-78b1-4d16-b1ed-d24aaeaf7543") failed: Property not found. retry in 59.936349ms file = illumos-utils/src/svc.rs:36 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49 zone = oxz_propolis-server_b926d0d2-78b1-4d16-b1ed-d24aaeaf7543
24862024-05-15T07:18:19.169ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_b926d0d2-78b1-4d16-b1ed-d24aaeaf7543") failed: Property not found. retry in 113.110852ms file = illumos-utils/src/svc.rs:36 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49 zone = oxz_propolis-server_b926d0d2-78b1-4d16-b1ed-d24aaeaf7543
24872024-05-15T07:18:19.300ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_b926d0d2-78b1-4d16-b1ed-d24aaeaf7543") failed: Property not found. retry in 577.037868ms file = illumos-utils/src/svc.rs:36 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49 zone = oxz_propolis-server_b926d0d2-78b1-4d16-b1ed-d24aaeaf7543
24882024-05-15T07:18:19.888ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_b926d0d2-78b1-4d16-b1ed-d24aaeaf7543") failed: Property not found. retry in 1.010831071s file = illumos-utils/src/svc.rs:36 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49 zone = oxz_propolis-server_b926d0d2-78b1-4d16-b1ed-d24aaeaf7543
24892024-05-15T07:18:20.837ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
24902024-05-15T07:18:20.909ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_b926d0d2-78b1-4d16-b1ed-d24aaeaf7543") failed: Property not found. retry in 1.036408746s file = illumos-utils/src/svc.rs:36 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49 zone = oxz_propolis-server_b926d0d2-78b1-4d16-b1ed-d24aaeaf7543
24912024-05-15T07:18:21.958ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_b926d0d2-78b1-4d16-b1ed-d24aaeaf7543") failed: Property not found. retry in 796.304506ms file = illumos-utils/src/svc.rs:36 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49 zone = oxz_propolis-server_b926d0d2-78b1-4d16-b1ed-d24aaeaf7543
24922024-05-15T07:18:22.767ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_b926d0d2-78b1-4d16-b1ed-d24aaeaf7543") failed: Property not found. retry in 715.342924ms file = illumos-utils/src/svc.rs:36 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49 zone = oxz_propolis-server_b926d0d2-78b1-4d16-b1ed-d24aaeaf7543
24932024-05-15T07:18:23.493ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_b926d0d2-78b1-4d16-b1ed-d24aaeaf7543") failed: Property not found. retry in 1.320581035s file = illumos-utils/src/svc.rs:36 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49 zone = oxz_propolis-server_b926d0d2-78b1-4d16-b1ed-d24aaeaf7543
24942024-05-15T07:18:24.841ZINFOSledAgent (InstanceManager): Started propolis in zone: oxz_propolis-server_b926d0d2-78b1-4d16-b1ed-d24aaeaf7543 file = sled-agent/src/instance.rs:1423 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49
24952024-05-15T07:18:24.849ZINFOSledAgent (InstanceManager): Propolis SMF service is online file = sled-agent/src/instance.rs:1432 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49
24962024-05-15T07:18:24.884ZINFOSledAgent (InstanceManager): Propolis HTTP server online file = sled-agent/src/instance.rs:1446 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49
24972024-05-15T07:18:24.886ZINFOSledAgent (InstanceManager): Sending ensure request to propolis: InstanceEnsureRequest { cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAAB6SDqvWK9YAABIOq9YAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAe0g6r1ivWAAASDqvWAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiI3NGI1ZDkzZS0yMjQxLTQzMTktODEyOS1mZTY3MTA4YzhmNDkiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJTDlyZFRFMkR3WXU5WGloTWxwdDQ4ZzU3OUhjZkROMDQ0MXpKTVhIbUFUZSJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"), disks: [DiskRequest { device: "nvme", name: "disk-7d72237e3cb", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: a2bdf7c7-d54c-4168-8d09-daf3c96c27c1, read_only_parent: Some(Volume { block_size: 512, id: 064e6874-7451-4713-8b9b-16ba9758e09b, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 98f450bf-ba2d-4a97-b47f-1d65c4edbb9e, key: Some("N/PJy4twLSmZ7nLWEfFW8oifHAzQ1XQV3m2X/kcGn88="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::13]:19001", "[fd00:1122:3344:101::12]:19001", "[fd00:1122:3344:101::19]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: a2bdf7c7-d54c-4168-8d09-daf3c96c27c1, key: Some("08OcIC7oIXvo3LKwtBjCBiI0Wxg52LMrRfXsu93NGs8="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::16]:19001", "[fd00:1122:3344:101::19]:19002", "[fd00:1122:3344:101::13]:19002"] } }] } }], migrate: None, nics: [NetworkInterfaceRequest { name: "vopte6", slot: Slot(0) }], properties: InstanceProperties { bootrom_id: 00000000-0000-0000-0000-000000000000, description: "Test description", id: 74b5d93e-2241-4319-8129-fe67108c8f49, image_id: 00000000-0000-0000-0000-000000000000, memory: 1024, metadata: InstanceMetadata { project_id: 22d38dba-1ffb-40fc-b3f3-4ee2a548bcf6, silo_id: 82ce2851-ee69-4933-b9a7-f0eee8dffe12 }, name: "localshark", vcpus: 2 } } file = sled-agent/src/instance.rs:685 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49
24982024-05-15T07:18:25.067ZINFOSledAgent (InstanceManager): result of instance_ensure call is Ok(InstanceEnsureResponse { migrate: None }) file = sled-agent/src/instance.rs:687 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49
24992024-05-15T07:18:25.069ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Starting), migration_status: NoMigration, time: 2024-05-15T07:18:25.069060454Z } file = sled-agent/src/instance.rs:569 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49
25002024-05-15T07:18:25.069ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:591 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49 new_instance_state = InstanceRuntimeState { propolis_id: Some(b926d0d2-78b1-4d16-b1ed-d24aaeaf7543), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-05-15T07:18:16.811647382Z } new_vmm_state = VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-05-15T07:18:25.069060454Z } propolis_id = b926d0d2-78b1-4d16-b1ed-d24aaeaf7543
25012024-05-15T07:18:25.069ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:491 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(b926d0d2-78b1-4d16-b1ed-d24aaeaf7543), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-05-15T07:18:16.811647382Z }, propolis_id: b926d0d2-78b1-4d16-b1ed-d24aaeaf7543, vmm_state: VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-05-15T07:18:25.069060454Z } }
25022024-05-15T07:18:25.127ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Running), migration_status: NoMigration, time: 2024-05-15T07:18:25.127457469Z } file = sled-agent/src/instance.rs:569 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49
25032024-05-15T07:18:25.127ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:591 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49 new_instance_state = InstanceRuntimeState { propolis_id: Some(b926d0d2-78b1-4d16-b1ed-d24aaeaf7543), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-05-15T07:18:16.811647382Z } new_vmm_state = VmmRuntimeState { state: Running, gen: Generation(3), time_updated: 2024-05-15T07:18:25.127457469Z } propolis_id = b926d0d2-78b1-4d16-b1ed-d24aaeaf7543
25042024-05-15T07:18:25.127ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:491 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(b926d0d2-78b1-4d16-b1ed-d24aaeaf7543), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-05-15T07:18:16.811647382Z }, propolis_id: b926d0d2-78b1-4d16-b1ed-d24aaeaf7543, vmm_state: VmmRuntimeState { state: Running, gen: Generation(3), time_updated: 2024-05-15T07:18:25.127457469Z } }
25052024-05-15T07:18:32.933ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50267
25062024-05-15T07:18:32.934ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 204 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50267 req_id = 40203884-97a7-466f-980a-3136923c76ce response_code = 200 uri = /instances/74b5d93e-2241-4319-8129-fe67108c8f49/state
25072024-05-15T07:18:33.028ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:64479
25082024-05-15T07:18:33.029ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 151 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:64479 req_id = ebc4ba71-5f07-419a-aebf-af82ff730189 response_code = 200 uri = /instances/74b5d93e-2241-4319-8129-fe67108c8f49/state
25092024-05-15T07:18:33.173ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:51655
25102024-05-15T07:18:33.173ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 72 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:51655 req_id = d8b25a24-acdc-42bf-8d59-e248dd96d12c response_code = 200 uri = /instances/74b5d93e-2241-4319-8129-fe67108c8f49/state
25112024-05-15T07:18:33.175ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:36263
25122024-05-15T07:18:33.223ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 48465 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:36263 req_id = 963148da-eb4b-4365-be4c-4f5dc58fae0f response_code = 204 uri = /switch-ports
25132024-05-15T07:18:33.224ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 638 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:36263 req_id = 7a5512d1-ee04-4237-9827-4104e81b6087 response_code = 200 uri = /network-bootstore-config
25142024-05-15T07:18:33.266ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62458
25152024-05-15T07:18:33.314ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 47563 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:62458 req_id = f5eb8092-9b65-4afd-9394-6f6b5f0fe954 response_code = 204 uri = /switch-ports
25162024-05-15T07:18:33.314ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 98 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:62458 req_id = 2da0135a-afee-4331-beea-1f0e2e92ee83 response_code = 200 uri = /network-bootstore-config
25172024-05-15T07:18:33.323ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:42679
25182024-05-15T07:18:33.370ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 46702 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:42679 req_id = 1111ecf7-a186-4f17-a519-f0387065f518 response_code = 204 uri = /switch-ports
25192024-05-15T07:18:33.370ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:42679 req_id = 3ef5cddd-cc5d-4746-a887-3d8760409416 response_code = 200 uri = /network-bootstore-config
25202024-05-15T07:18:33.933ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:55095
25212024-05-15T07:18:33.933ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 71 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:55095 req_id = fb108a3e-8f3c-4e3d-aed5-7d261f118d40 response_code = 200 uri = /baseboard
25222024-05-15T07:18:34.845ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59545
25232024-05-15T07:18:34.845ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 606 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:59545 req_id = b9df7b19-4860-482a-b149-1f31e8f09abd response_code = 200 uri = /instances/74b5d93e-2241-4319-8129-fe67108c8f49/state
25242024-05-15T07:18:34.845ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Stopping), migration_status: NoMigration, time: 2024-05-15T07:18:34.845824263Z } file = sled-agent/src/instance.rs:569 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49
25252024-05-15T07:18:34.845ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:591 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49 new_instance_state = InstanceRuntimeState { propolis_id: Some(b926d0d2-78b1-4d16-b1ed-d24aaeaf7543), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-05-15T07:18:16.811647382Z } new_vmm_state = VmmRuntimeState { state: Stopping, gen: Generation(5), time_updated: 2024-05-15T07:18:34.845824263Z } propolis_id = b926d0d2-78b1-4d16-b1ed-d24aaeaf7543
25262024-05-15T07:18:34.845ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:491 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(b926d0d2-78b1-4d16-b1ed-d24aaeaf7543), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-05-15T07:18:16.811647382Z }, propolis_id: b926d0d2-78b1-4d16-b1ed-d24aaeaf7543, vmm_state: VmmRuntimeState { state: Stopping, gen: Generation(5), time_updated: 2024-05-15T07:18:34.845824263Z } }
25272024-05-15T07:18:34.891ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Destroyed), migration_status: NoMigration, time: 2024-05-15T07:18:34.891896712Z } file = sled-agent/src/instance.rs:569 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49
25282024-05-15T07:18:34.891ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:591 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49 new_instance_state = InstanceRuntimeState { propolis_id: None, dst_propolis_id: None, migration_id: None, gen: Generation(5), time_updated: 2024-05-15T07:18:34.891896712Z } new_vmm_state = VmmRuntimeState { state: Destroyed, gen: Generation(6), time_updated: 2024-05-15T07:18:34.891896712Z } propolis_id = b926d0d2-78b1-4d16-b1ed-d24aaeaf7543
25292024-05-15T07:18:34.891ZINFOSledAgent (InstanceManager): terminating VMM that has exited file = sled-agent/src/instance.rs:606 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49
25302024-05-15T07:18:34.891ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
25312024-05-15T07:18:34.892ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
25322024-05-15T07:18:34.892ZINFOSledAgent (ZoneBundler): creating zone bundle context = ZoneBundleContext { storage_dirs: ["/pool/int/3907cd6d-3c6d-4543-92de-9ec01f9bc32c/debug/bundle/zone", "/pool/int/c3ed08c3-e0e1-43f2-b040-3c4755a2167c/debug/bundle/zone"], cause: TerminatedInstance, extra_log_dirs: ["/pool/ext/4e6ef30f-a7d9-4acc-99aa-59ae1084fa14/crypt/debug", "/pool/ext/12d428b9-eedd-40c0-8928-1d92a73f181d/crypt/debug", "/pool/ext/08d4507c-e72e-4afd-9f03-5e28f4ad0909/crypt/debug", "/pool/ext/7ba4af02-bf8f-47c6-af79-d69fdad5d4bf/crypt/debug", "/pool/ext/62a187a2-0706-420e-a881-83ff781ad6c1/crypt/debug", "/pool/ext/5da68ecb-08f6-4895-925f-903c81000441/crypt/debug", "/pool/ext/3b36f3f1-29d0-4d11-8b6b-ae2a89620d43/crypt/debug", "/pool/ext/2780ea55-1799-4c3e-916a-9024b9586eaf/crypt/debug", "/pool/ext/bae019e4-4e95-466e-aa8a-f660660e26c1/crypt/debug"] } file = sled-agent/src/zone_bundle.rs:437 zone_name = oxz_propolis-server_b926d0d2-78b1-4d16-b1ed-d24aaeaf7543
25332024-05-15T07:18:35.531ZINFOSledAgent (ZoneBundler): finished zone bundle file = sled-agent/src/zone_bundle.rs:1230 metadata = ZoneBundleMetadata { id: ZoneBundleId { zone_name: "oxz_propolis-server_b926d0d2-78b1-4d16-b1ed-d24aaeaf7543", bundle_id: 54a3329a-dfe0-4d98-904a-0afc9d2e7667 }, time_created: 2024-05-15T07:18:34.892497534Z, version: 0, cause: TerminatedInstance }
25342024-05-15T07:18:35.531ZWARNSledAgent (InstanceManager): Halting and removing zone: oxz_propolis-server_b926d0d2-78b1-4d16-b1ed-d24aaeaf7543 file = sled-agent/src/instance.rs:780 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49
25352024-05-15T07:18:37.284ZINFOSledAgent (InstanceManager): halt_and_remove_logged: Previous zone state: Running file = illumos-utils/src/zone.rs:269 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49
25362024-05-15T07:18:37.312ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:491 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: None, dst_propolis_id: None, migration_id: None, gen: Generation(5), time_updated: 2024-05-15T07:18:34.891896712Z }, propolis_id: b926d0d2-78b1-4d16-b1ed-d24aaeaf7543, vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-05-15T07:18:37.312163043Z } }
25372024-05-15T07:18:37.388ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:50316
25382024-05-15T07:18:37.390ZWARNSledAgent (PortManager): unset_virtual_nic_host unimplmented file = illumos-utils/src/opte/port_manager.rs:620
25392024-05-15T07:18:37.390ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 1698 local_addr = [fd00:1122:3344:101::1]:12345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:50316 req_id = e909a894-9f95-4ab3-8e6b-b5686c5b6344 response_code = 204 uri = /v2p/6a12a6a1-9861-4fe8-bb03-f96ecb313186
25402024-05-15T07:18:37.650ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:491 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: None, dst_propolis_id: None, migration_id: None, gen: Generation(5), time_updated: 2024-05-15T07:18:34.891896712Z }, propolis_id: b926d0d2-78b1-4d16-b1ed-d24aaeaf7543, vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-05-15T07:18:37.312163043Z } }
25412024-05-15T07:18:37.654ZINFOSledAgent (InstanceManager): State monitoring task complete file = sled-agent/src/instance.rs:723 instance_id = 74b5d93e-2241-4319-8129-fe67108c8f49