Max level shown:
1 [ May 14 09:39:02 Enabled. ]
2 [ May 14 09:39:02 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]
3 [ May 14 09:39:02 Method "start" exited with status 0. ]
4 [ May 14 09:39:02 Rereading configuration. ]
5 [ May 14 09:39:02 No 'refresh' method defined. Treating as :true. ]
6 note: configured to log to "/dev/stdout"
72024-05-14T09:39:02.796ZINFOSledAgent: Ensuring contract reaper thread file = illumos-utils/src/running_zone.rs:163
82024-05-14T09:39:02.796ZINFOSledAgent (BootstrapAgentStartup): finding links ["net0", "net1"] file = sled-agent/src/bootstrap/pre_server.rs:255
92024-05-14T09:39:02.853ZINFOSledAgent (BootstrapAgentStartup): Starting mg-ddm service addr-objs = [AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }] file = sled-agent/src/bootstrap/pre_server.rs:262
102024-05-14T09:39:02.853ZINFOSledAgent (BootstrapAgentStartup): Importing mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:42 path = /opt/oxide/mg-ddm/pkg/ddm/manifest.xml
112024-05-14T09:39:02.969ZINFOSledAgent (BootstrapAgentStartup): Setting mg-ddm interfaces file = sled-agent/src/bootstrap/maghemite.rs:50 interfaces = ("net0/ll" "net1/ll")
122024-05-14T09:39:02.978ZINFOSledAgent (BootstrapAgentStartup): Enabling mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:56
132024-05-14T09:39:02.986ZINFOSledAgent (BootstrapAgentStartup): Ensuring zfs key directory exists file = sled-agent/src/bootstrap/pre_server.rs:272 path = /var/run/oxide/
142024-05-14T09:39:03.588ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:87 prefix = Ipv6Prefix { addr: fdb0:18c0:4d0d:a02a::, len: 64 }
152024-05-14T09:39:03.725ZINFOSledAgent: Starting KeyManager file = sled-agent/src/long_running_tasks.rs:119
162024-05-14T09:39:03.725ZINFOSledAgent: Starting StorageManager file = sled-agent/src/long_running_tasks.rs:131
172024-05-14T09:39:03.725ZINFOSledAgent: Starting StorageMonitor file = sled-agent/src/long_running_tasks.rs:141
182024-05-14T09:39:03.725ZINFOSledAgent: Starting HardwareManager file = sled-agent/src/long_running_tasks.rs:162 nongimlet_observed_disks = [] sled_mode = Scrimlet { asic: SoftNpuZone }
192024-05-14T09:39:03.725ZINFOSledAgent (DumpSetup-archival): DumpSetup poll loop started. file = sled-agent/src/dump_setup.rs:291
202024-05-14T09:39:03.725ZINFOSledAgent (HardwareManager): Creating HardwareManager file = sled-hardware/src/illumos/mod.rs:625
212024-05-14T09:39:03.725ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:801
222024-05-14T09:39:03.728ZINFOSledAgent (HardwareManager): Generated i86pc baseboard Pc { identifier: "sock", model: "i86pc" } file = sled-hardware/src/illumos/mod.rs:535
232024-05-14T09:39:03.728ZWARNSledAgent (HardwareManager): Device is not a Gimlet (i86pc), proceeding with null hardware view file = sled-hardware/src/illumos/mod.rs:674
242024-05-14T09:39:03.728ZINFOSledAgent: Starting HardwareMonitor file = sled-agent/src/long_running_tasks.rs:176
252024-05-14T09:39:03.728ZINFOSledAgent (HardwareManager): Monitoring for hardware updates file = sled-hardware/src/illumos/mod.rs:733
262024-05-14T09:39:03.728ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/m2_0.vdev
272024-05-14T09:39:03.728ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:262 underlay_network_info = None
282024-05-14T09:39:03.728ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }), tx: ... } file = sled-storage/src/manager.rs:381
292024-05-14T09:39:03.728ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }
302024-05-14T09:39:03.728ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:112 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" } path = /scratch/m2_0.vdev
312024-05-14T09:39:03.741ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_0.vdev file = sled-hardware/src/disk.rs:304
322024-05-14T09:39:03.741ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:315 id = 112d7827-193f-456c-94a5-19163d29475e (zpool)
332024-05-14T09:39:03.842ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 112d7827-193f-456c-94a5-19163d29475e (zpool), kind: Internal }
342024-05-14T09:39:04.044ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:381
352024-05-14T09:39:04.044ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/m2_1.vdev
362024-05-14T09:39:04.044ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }
372024-05-14T09:39:04.044ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }
382024-05-14T09:39:04.044ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }), tx: ... } file = sled-storage/src/manager.rs:381
392024-05-14T09:39:04.044ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }
402024-05-14T09:39:04.044ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:112 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" } path = /scratch/m2_1.vdev
412024-05-14T09:39:04.053ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_1.vdev file = sled-hardware/src/disk.rs:304
422024-05-14T09:39:04.053ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:315 id = e21e7560-2c98-46a4-bdda-f772e50ec2a3 (zpool)
432024-05-14T09:39:04.142ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: e21e7560-2c98-46a4-bdda-f772e50ec2a3 (zpool), kind: Internal }
442024-05-14T09:39:04.356ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_0.vdev
452024-05-14T09:39:04.356ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 112d7827-193f-456c-94a5-19163d29475e (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: e21e7560-2c98-46a4-bdda-f772e50ec2a3 (zpool), kind: Internal } }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
462024-05-14T09:39:04.356ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 }), tx: ... } file = sled-storage/src/manager.rs:381
472024-05-14T09:39:04.356ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }
482024-05-14T09:39:04.356ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_1.vdev
492024-05-14T09:39:04.356ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 112d7827-193f-456c-94a5-19163d29475e (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: e21e7560-2c98-46a4-bdda-f772e50ec2a3 (zpool), kind: Internal } }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
502024-05-14T09:39:04.356ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 }), tx: ... } file = sled-storage/src/manager.rs:381
512024-05-14T09:39:04.356ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }
522024-05-14T09:39:04.356ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_2.vdev
532024-05-14T09:39:04.356ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 }), tx: ... } file = sled-storage/src/manager.rs:381
542024-05-14T09:39:04.356ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 112d7827-193f-456c-94a5-19163d29475e (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: e21e7560-2c98-46a4-bdda-f772e50ec2a3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
552024-05-14T09:39:04.356ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }
562024-05-14T09:39:04.356ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_3.vdev
572024-05-14T09:39:04.356ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 112d7827-193f-456c-94a5-19163d29475e (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: e21e7560-2c98-46a4-bdda-f772e50ec2a3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
582024-05-14T09:39:04.356ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 }), tx: ... } file = sled-storage/src/manager.rs:381
592024-05-14T09:39:04.356ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }
602024-05-14T09:39:04.356ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_4.vdev
612024-05-14T09:39:04.356ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, variant: U2, slot: 1030 }), tx: ... } file = sled-storage/src/manager.rs:381
622024-05-14T09:39:04.356ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }
632024-05-14T09:39:04.356ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 112d7827-193f-456c-94a5-19163d29475e (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: e21e7560-2c98-46a4-bdda-f772e50ec2a3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
642024-05-14T09:39:04.356ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_5.vdev
652024-05-14T09:39:04.356ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, variant: U2, slot: 1031 }), tx: ... } file = sled-storage/src/manager.rs:381
662024-05-14T09:39:04.356ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 112d7827-193f-456c-94a5-19163d29475e (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: e21e7560-2c98-46a4-bdda-f772e50ec2a3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, variant: U2, slot: 1030 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
672024-05-14T09:39:04.356ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }
682024-05-14T09:39:04.356ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_6.vdev
692024-05-14T09:39:04.356ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }, variant: U2, slot: 1032 }), tx: ... } file = sled-storage/src/manager.rs:381
702024-05-14T09:39:04.356ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }
712024-05-14T09:39:04.356ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 112d7827-193f-456c-94a5-19163d29475e (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: e21e7560-2c98-46a4-bdda-f772e50ec2a3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, variant: U2, slot: 1030 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, variant: U2, slot: 1031 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
722024-05-14T09:39:04.356ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_7.vdev
732024-05-14T09:39:04.356ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }, variant: U2, slot: 1033 }), tx: ... } file = sled-storage/src/manager.rs:381
742024-05-14T09:39:04.356ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }
752024-05-14T09:39:04.356ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 112d7827-193f-456c-94a5-19163d29475e (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: e21e7560-2c98-46a4-bdda-f772e50ec2a3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, variant: U2, slot: 1030 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, variant: U2, slot: 1031 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }, variant: U2, slot: 1032 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
762024-05-14T09:39:04.356ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_8.vdev
772024-05-14T09:39:04.356ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }, variant: U2, slot: 1034 }), tx: ... } file = sled-storage/src/manager.rs:381
782024-05-14T09:39:04.356ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }
792024-05-14T09:39:04.356ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 112d7827-193f-456c-94a5-19163d29475e (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: e21e7560-2c98-46a4-bdda-f772e50ec2a3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, variant: U2, slot: 1030 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, variant: U2, slot: 1031 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }, variant: U2, slot: 1032 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }, variant: U2, slot: 1033 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
802024-05-14T09:39:04.356ZINFOSledAgent: Waiting for boot disk file = sled-agent/src/long_running_tasks.rs:91
812024-05-14T09:39:04.356ZINFOSledAgent: Found boot disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" } file = sled-agent/src/long_running_tasks.rs:93
822024-05-14T09:39:04.356ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 112d7827-193f-456c-94a5-19163d29475e (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: e21e7560-2c98-46a4-bdda-f772e50ec2a3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, variant: U2, slot: 1030 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, variant: U2, slot: 1031 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }, variant: U2, slot: 1032 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }, variant: U2, slot: 1033 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }, variant: U2, slot: 1034 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
832024-05-14T09:39:04.356ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
842024-05-14T09:39:04.356ZINFOSledAgent: Starting Bootstore file = sled-agent/src/long_running_tasks.rs:200
852024-05-14T09:39:04.356ZINFOSledAgent (bootstore): No ledger in /pool/int/112d7827-193f-456c-94a5-19163d29475e/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
862024-05-14T09:39:04.356ZINFOSledAgent (bootstore): No ledger in /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
872024-05-14T09:39:04.356ZINFOSledAgent (bootstore): No ledger found. Loading Fsm::State as Uninitialized file = bootstore/src/schemes/v0/storage.rs:83 peer_id = pc-sock-i86pc
882024-05-14T09:39:04.356ZINFOSledAgent (bootstore): No ledger in /pool/int/112d7827-193f-456c-94a5-19163d29475e/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
892024-05-14T09:39:04.356ZINFOSledAgent (bootstore): No ledger in /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
902024-05-14T09:39:04.356ZINFOSledAgent (bootstore): No ledger found for network config file = bootstore/src/schemes/v0/storage.rs:146 peer_id = pc-sock-i86pc
912024-05-14T09:39:04.356ZINFOSledAgent: Starting Bootstore DDMD poller file = sled-agent/src/long_running_tasks.rs:205
922024-05-14T09:39:04.356ZINFOSledAgent: Starting ZoneBundler related tasks file = sled-agent/src/long_running_tasks.rs:220
932024-05-14T09:39:04.367ZINFOSledAgent (ServiceManager): Creating ServiceManager file = sled-agent/src/services.rs:711
942024-05-14T09:39:04.367ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:289 next_cleanup = Instant { tv_sec: 1353, tv_nsec: 439613001 } time_to_next_cleanup = 599.999944149s
952024-05-14T09:39:04.367ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
962024-05-14T09:39:04.367ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3617
972024-05-14T09:39:04.367ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/sled-agent-request.json file = common/src/ledger.rs:188
982024-05-14T09:39:04.367ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/sled-agent-request.json file = common/src/ledger.rs:188
992024-05-14T09:39:04.371ZINFOSledAgent (dropshot (BootstrapAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:196 local_addr = [fdb0:18c0:4d0d:a02a::1]:80
1002024-05-14T09:39:04.371ZINFOSledAgent (SledAgentSprocketsServer): Started listening file = sled-agent/src/bootstrap/sprockets_server.rs:44 local_addr = [fdb0:18c0:4d0d:a02a::1]:12346
1012024-05-14T09:39:04.371ZINFOSledAgent (RSS): Injecting RSS configuration: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.2.164, last: 10.151.2.173, }, ), ], external_dns_ips: [ 10.151.2.164, 10.151.2.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV1 { rack_subnet: Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56, }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV1 { routes: [ RouteConfig { destination: V4( Ipv4Network { addr: 0.0.0.0, prefix: 0, }, ), nexthop: 10.151.2.100, vlan_id: None, }, ], addresses: [ V4( Ipv4Network { addr: 10.151.2.174, prefix: 24, }, ), ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, } file = sled-agent/src/rack_setup/service.rs:1003
1022024-05-14T09:39:04.371ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
1032024-05-14T09:39:04.371ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1042024-05-14T09:39:04.371ZINFOSledAgent (RSS): No ledger in /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1052024-05-14T09:39:04.371ZINFOSledAgent (RSS): No ledger in /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1062024-05-14T09:39:04.371ZINFOSledAgent (RSS): RSS configuration has not been fully applied yet file = sled-agent/src/rack_setup/service.rs:1063
1072024-05-14T09:39:04.371ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1082024-05-14T09:39:04.371ZINFOSledAgent (RSS): No ledger in /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/rss-sled-plan.json file = common/src/ledger.rs:188
1092024-05-14T09:39:04.371ZINFOSledAgent (RSS): No ledger in /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/rss-sled-plan.json file = common/src/ledger.rs:188
1102024-05-14T09:39:04.371ZINFOSledAgent (RSS): Creating new allocation plan file = sled-agent/src/rack_setup/service.rs:1104
1112024-05-14T09:39:04.371ZINFOSledAgent (RSS): Serializing plan file = sled-agent/src/rack_setup/plan/sled.rs:118
1122024-05-14T09:39:04.371ZINFOSledAgent (RSS): Creating plan for the sled at fdb0:18c0:4d0d:a02a::1 file = sled-agent/src/rack_setup/plan/sled.rs:91
1132024-05-14T09:39:04.371ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1142024-05-14T09:39:04.371ZINFOSledAgent (RSS): Writing ledger to /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1152024-05-14T09:39:04.372ZINFOSledAgent (RSS): Writing ledger to /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1162024-05-14T09:39:04.372ZINFOSledAgent (RSS): Sled plan written to storage file = sled-agent/src/rack_setup/plan/sled.rs:138
1172024-05-14T09:39:04.372ZINFOSledAgent (RSS): Writing Rack Network Configuration to bootstore file = sled-agent/src/rack_setup/service.rs:1137
1182024-05-14T09:39:04.372ZINFOSledAgent (bootstore): Attempting to update network config with generation: 1, current_generation: 0 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-sock-i86pc
1192024-05-14T09:39:04.372ZINFOSledAgent (bootstore): Writing ledger to /pool/int/112d7827-193f-456c-94a5-19163d29475e/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
1202024-05-14T09:39:04.372ZINFOSledAgent (bootstore): Writing ledger to /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
1212024-05-14T09:39:04.372ZINFOSledAgent (bootstore): Broadcasting network config with generation 1 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-sock-i86pc
1222024-05-14T09:39:04.372ZINFOSledAgent (BootstrapAgentRssHandler): Received initialization request from RSS file = sled-agent/src/bootstrap/rss_handle.rs:217 request = StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 5f617488-2ef8-4cf1-81f7-2b249f5b34c0, rack_id: 240cabb9-b684-4172-8606-d31cd2279cd6, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net(Ipv6Network { addr: fd00:1122:3344:101::, prefix: 64 }) } } } target_sled = [fdb0:18c0:4d0d:a02a::1]:12346
1232024-05-14T09:39:04.372ZINFOSledAgent (SledAgentSprocketsServer): Accepted connection file = sled-agent/src/bootstrap/sprockets_server.rs:66 remote_addr = [fdb0:18c0:4d0d:a02a::1]:34794
1242024-05-14T09:39:04.372ZINFOSledAgent (SledAgentMain): Loading Sled Agent: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 5f617488-2ef8-4cf1-81f7-2b249f5b34c0, rack_id: 240cabb9-b684-4172-8606-d31cd2279cd6, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net(Ipv6Network { addr: fd00:1122:3344:101::, prefix: 64 }) } } } file = sled-agent/src/bootstrap/server.rs:360
1252024-05-14T09:39:04.372ZINFOSledAgent (SledAgentMain): KeyManager: using hardcoded secret retriever file = sled-agent/src/bootstrap/server.rs:375
1262024-05-14T09:39:04.372ZINFOSledAgent: setting up sled agent server file = sled-agent/src/server.rs:44
1272024-05-14T09:39:04.373ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
1282024-05-14T09:39:04.373ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:87 prefix = Ipv6Prefix { addr: fd00:1122:3344:101::, len: 64 }
1292024-05-14T09:39:04.373ZINFOSledAgent (StorageManager): Received KeyManagerReady file = sled-storage/src/manager.rs:381
1302024-05-14T09:39:04.373ZINFOSledAgent (StorageManager): Loading storage config file = sled-storage/src/manager.rs:530
1312024-05-14T09:39:04.373ZINFOSledAgent (StorageManager): No ledger in /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_ledger
1322024-05-14T09:39:04.373ZINFOSledAgent (StorageManager): No ledger in /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_ledger
1332024-05-14T09:39:04.373ZINFOSledAgent (StorageManager): No ledger of physical disks exists file = sled-storage/src/manager.rs:519
1342024-05-14T09:39:04.373ZINFOSledAgent (StorageManager): KeyManager ready, but no ledger detected file = sled-storage/src/manager.rs:547
1352024-05-14T09:39:04.376ZINFOSledAgent (ServiceManager): Enabling switch zone (new) file = sled-agent/src/services.rs:3849
1362024-05-14T09:39:04.385ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }
1372024-05-14T09:39:04.395ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }
1382024-05-14T09:39:04.406ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }
1392024-05-14T09:39:04.415ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }
1402024-05-14T09:39:04.424ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }
1412024-05-14T09:39:04.433ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }
1422024-05-14T09:39:04.442ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }
1432024-05-14T09:39:04.451ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }
1442024-05-14T09:39:04.460ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }
1452024-05-14T09:39:04.460ZINFOSledAgent (StorageManager): No disks to be automatically managed file = sled-storage/src/manager.rs:558
1462024-05-14T09:39:04.460ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1472024-05-14T09:39:04.460ZINFOSledAgent (StorageManager): automatically managing disks file = sled-storage/src/manager.rs:370
1482024-05-14T09:39:04.460ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:339
1492024-05-14T09:39:04.460ZINFOSledAgent (StorageManager): Successfully synchronized disks without error file = sled-storage/src/manager.rs:449 result = DisksManagementResult { status: [] }
1502024-05-14T09:39:04.470ZINFOSledAgent: SledAgent::new(..) starting file = sled-agent/src/sled_agent.rs:355 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
1512024-05-14T09:39:04.470ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1522024-05-14T09:39:04.470ZINFOSledAgent: Requested swap device of size 64 GiB file = sled-agent/src/sled_agent.rs:367 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
1532024-05-14T09:39:04.471ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1542024-05-14T09:39:04.473ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 135.925954ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "3f8a4438-05b7-48c6-aa43-d39fa518992c", "content-length": "124", "date": "Tue, 14 May 2024 09:39:03 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "3f8a4438-05b7-48c6-aa43-d39fa518992c" } file = clients/ddm-admin-client/src/lib.rs:178
1552024-05-14T09:39:04.481ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:315
1562024-05-14T09:39:04.484ZINFOSledAgent: attempting to create encrypted zvol: name="oxi_112d7827-193f-456c-94a5-19163d29475e/swap", size_gb=64 file = sled-agent/src/swap_device.rs:153
1572024-05-14T09:39:04.499ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:354
1582024-05-14T09:39:04.506ZINFOSledAgent: successfully created encrypted zvol: name="oxi_112d7827-193f-456c-94a5-19163d29475e/swap", size_gb=64 file = sled-agent/src/swap_device.rs:249
1592024-05-14T09:39:04.506ZINFOSledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_112d7827-193f-456c-94a5-19163d29475e/swap" file = sled-agent/src/swap_device.rs:96
1602024-05-14T09:39:04.511ZINFOSledAgent: Mounting backing filesystems file = sled-agent/src/sled_agent.rs:382 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
1612024-05-14T09:39:04.511ZINFOSledAgent (BackingFs): Processing fmd file = sled-agent/src/backing_fs.rs:126
1622024-05-14T09:39:04.511ZINFOSledAgent (BackingFs): Ensuring dataset oxi_112d7827-193f-456c-94a5-19163d29475e/backing/fmd file = sled-agent/src/backing_fs.rs:136
1632024-05-14T09:39:04.581ZINFOSledAgent (BackingFs): Stopping service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:170
1642024-05-14T09:39:04.610ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1652024-05-14T09:39:04.610ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 107.535102ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "ce9370f9-ff03-43f8-81d1-33ed389a2d63", "content-length": "124", "date": "Tue, 14 May 2024 09:39:04 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "ce9370f9-ff03-43f8-81d1-33ed389a2d63" } file = clients/ddm-admin-client/src/lib.rs:178
1662024-05-14T09:39:04.719ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1672024-05-14T09:39:04.720ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 110.970552ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "26401e8b-09e1-4e0b-b8ec-b5f724359197", "content-length": "124", "date": "Tue, 14 May 2024 09:39:04 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "26401e8b-09e1-4e0b-b8ec-b5f724359197" } file = clients/ddm-admin-client/src/lib.rs:178
1682024-05-14T09:39:04.801ZINFOSledAgent (BackingFs): Mounting oxi_112d7827-193f-456c-94a5-19163d29475e/backing/fmd on /var/fm/fmd file = sled-agent/src/backing_fs.rs:178
1692024-05-14T09:39:04.815ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/rsrc file = sled-agent/src/backing_fs.rs:186
1702024-05-14T09:39:04.815ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/ckpt file = sled-agent/src/backing_fs.rs:186
1712024-05-14T09:39:04.815ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/xprt file = sled-agent/src/backing_fs.rs:186
1722024-05-14T09:39:04.815ZINFOSledAgent (BackingFs): Starting service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:192
1732024-05-14T09:39:04.832ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1742024-05-14T09:39:04.833ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 95.368922ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "2eab517e-6bc3-4bb9-87ca-3f02aaae61e2", "content-length": "124", "date": "Tue, 14 May 2024 09:39:04 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "2eab517e-6bc3-4bb9-87ca-3f02aaae61e2" } file = clients/ddm-admin-client/src/lib.rs:178
1752024-05-14T09:39:04.929ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1762024-05-14T09:39:04.930ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 286.030246ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "6e8b1c31-30bc-4c63-8b1e-1c6034b901f0", "content-length": "124", "date": "Tue, 14 May 2024 09:39:04 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "6e8b1c31-30bc-4c63-8b1e-1c6034b901f0" } file = clients/ddm-admin-client/src/lib.rs:178
1772024-05-14T09:39:05.217ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1782024-05-14T09:39:05.218ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 358.09952ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "17c2930c-2214-4797-9a39-7c5ba3edb47b", "content-length": "124", "date": "Tue, 14 May 2024 09:39:04 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "17c2930c-2214-4797-9a39-7c5ba3edb47b" } file = clients/ddm-admin-client/src/lib.rs:178
1792024-05-14T09:39:05.567ZINFOSledAgent: using '[AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }]' as data links for xde driver file = illumos-utils/src/opte/illumos.rs:90 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
1802024-05-14T09:39:05.568ZINFOSledAgent (dropshot): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:196 local_addr = [fd00:1122:3344:101::1]:51153 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
1812024-05-14T09:39:05.568ZINFOSledAgent (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
1822024-05-14T09:39:05.568ZINFOSledAgent (producer-server): starting oximeter metric producer server address = [fd00:1122:3344:101::1]:51153 file = oximeter/producer/src/lib.rs:284 interval = 30s producer_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0 route = //5f617488-2ef8-4cf1-81f7-2b249f5b34c0 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
1832024-05-14T09:39:05.576ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 231.015917ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:393 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
1842024-05-14T09:39:05.579ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1852024-05-14T09:39:05.622ZINFOSledAgent (VmmReservoirManager): 50% of 102952026112 physical ram = 51476013056 bytes) file = sled-agent/src/vmm_reservoir.rs:267 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
1862024-05-14T09:39:05.622ZINFOSledAgent (VmmReservoirManager): Setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:275 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
1872024-05-14T09:39:05.622ZINFOSledAgent (ServiceManager): sled agent started file = sled-agent/src/services.rs:993 underlay_address = fd00:1122:3344:101::1
1882024-05-14T09:39:05.622ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
1892024-05-14T09:39:05.623ZINFOSledAgent (dropshot (SledAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:196 local_addr = [fd00:1122:3344:101::1]:12345
1902024-05-14T09:39:05.623ZINFOSledAgent (SledAgentMain): Sled Agent loaded; recording configuration file = sled-agent/src/bootstrap/server.rs:430
1912024-05-14T09:39:05.623ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1922024-05-14T09:39:05.623ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
1932024-05-14T09:39:05.623ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
1942024-05-14T09:39:05.624ZINFOSledAgent (HardwareMonitor): Sled Agent Started file = sled-agent/src/hardware_monitor.rs:138
1952024-05-14T09:39:05.624ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:262 underlay_network_info = Some((fd00:1122:3344:101::2, Some(RackNetworkConfigV1 { rack_subnet: Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56 }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [PortConfigV1 { routes: [RouteConfig { destination: V4(Ipv4Network { addr: 0.0.0.0, prefix: 0 }), nexthop: 10.151.2.100, vlan_id: None }], addresses: [V4(Ipv4Network { addr: 10.151.2.174, prefix: 24 })], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false }], bgp: [], bfd: [] })))
1962024-05-14T09:39:05.624ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3617
1972024-05-14T09:39:05.624ZINFOSledAgent (SledAgentSprocketsServer): Connection closed file = sled-agent/src/bootstrap/sprockets_server.rs:73 remote_addr = [fdb0:18c0:4d0d:a02a::1]:34794
1982024-05-14T09:39:05.848ZINFOSledAgent (BootstrapAgentRssHandler): Peer agent initialized file = sled-agent/src/bootstrap/rss_handle.rs:108 peer = [fdb0:18c0:4d0d:a02a::1]:12346
1992024-05-14T09:39:05.848ZINFOSledAgent (BootstrapAgentRssHandler): Initialized sled agent file = sled-agent/src/bootstrap/rss_handle.rs:232 target_sled = [fdb0:18c0:4d0d:a02a::1]:12346
2002024-05-14T09:39:05.848ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2012024-05-14T09:39:05.848ZINFOSledAgent (RSS): No ledger in /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/rss-service-plan-v3.json file = common/src/ledger.rs:188
2022024-05-14T09:39:05.848ZINFOSledAgent (RSS): No ledger in /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/rss-service-plan-v3.json file = common/src/ledger.rs:188
2032024-05-14T09:39:05.848ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2042024-05-14T09:39:05.848ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2052024-05-14T09:39:05.850ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 506.261316ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:393 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
2062024-05-14T09:39:05.853ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/5f617488-2ef8-4cf1-81f7-2b249f5b34c0): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
2072024-05-14T09:39:05.881ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:54427
2082024-05-14T09:39:05.881ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2092024-05-14T09:39:05.881ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 151 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:54427 req_id = c26931b0-b45a-44c4-b668-265879ba7244 response_code = 200 uri = /inventory
2102024-05-14T09:39:05.911ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:47178
2112024-05-14T09:39:05.911ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 46 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:47178 req_id = d0494a80-b9c0-4032-b7aa-519d88661755 response_code = 200 uri = /sled-role
2122024-05-14T09:39:05.912ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2132024-05-14T09:39:05.912ZINFOSledAgent (RSS): Writing ledger to /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/.rss-service-plan-v3.json.tmp file = common/src/ledger.rs:199
2142024-05-14T09:39:05.912ZINFOSledAgent (RSS): Writing ledger to /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/.rss-service-plan-v3.json.tmp file = common/src/ledger.rs:199
2152024-05-14T09:39:05.912ZINFOSledAgent (RSS): Service plan written to storage file = sled-agent/src/rack_setup/plan/service.rs:815
2162024-05-14T09:39:05.941ZINFOSledAgent (RSS): attempting to set up sled's storage: OmicronPhysicalDisksConfig { disks: [OmicronPhysicalDiskConfig { id: ea952c80-c23a-4ee1-bd80-680243d16d24, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, pool_id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool) }, OmicronPhysicalDiskConfig { id: 14f66ddc-1eb5-48be-b89b-a8fa9533ecf1, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, pool_id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool) }, OmicronPhysicalDiskConfig { id: 725c27a8-a484-4e46-8501-1dfe7e2922bc, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, pool_id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool) }, OmicronPhysicalDiskConfig { id: 88a38c3a-e1c0-4044-94de-a3eb481ddab6, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, pool_id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool) }, OmicronPhysicalDiskConfig { id: efc9f2a3-35bf-41bc-a0ef-d41e2887e1f6, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, pool_id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool) }, OmicronPhysicalDiskConfig { id: 71dc3281-b536-40e6-adbc-653e228e1156, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, pool_id: ae2f89f7-35c8-4c05-940c-cdee6d8f8a12 (zpool) }, OmicronPhysicalDiskConfig { id: f3ab4730-e13b-472f-b5bc-86c0264d5763, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }, pool_id: 42b2209d-f16a-4fca-a84d-54ecc9f13877 (zpool) }, OmicronPhysicalDiskConfig { id: 5cb4d019-d265-4039-9f26-1b59e0c7533d, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }, pool_id: 95ce52af-b115-45d4-acd1-3598b0bafa5d (zpool) }, OmicronPhysicalDiskConfig { id: b7371311-29d9-4467-886d-e4e8f1bc545e, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }, pool_id: ad348a98-27df-494e-af4b-560a8a421d64 (zpool) }], generation: Generation(1) } file = sled-agent/src/rack_setup/service.rs:347 sled_address = [fd00:1122:3344:101::1]:12345
2172024-05-14T09:39:05.942ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:55076
2182024-05-14T09:39:05.942ZINFOSledAgent (StorageManager): Received OmicronPhysicalDisksEnsure { config: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, id: ea952c80-c23a-4ee1-bd80-680243d16d24, pool_id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, id: 14f66ddc-1eb5-48be-b89b-a8fa9533ecf1, pool_id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, id: 725c27a8-a484-4e46-8501-1dfe7e2922bc, pool_id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, id: 88a38c3a-e1c0-4044-94de-a3eb481ddab6, pool_id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, id: efc9f2a3-35bf-41bc-a0ef-d41e2887e1f6, pool_id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, id: 71dc3281-b536-40e6-adbc-653e228e1156, pool_id: ae2f89f7-35c8-4c05-940c-cdee6d8f8a12 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }, id: f3ab4730-e13b-472f-b5bc-86c0264d5763, pool_id: 42b2209d-f16a-4fca-a84d-54ecc9f13877 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }, id: 5cb4d019-d265-4039-9f26-1b59e0c7533d, pool_id: 95ce52af-b115-45d4-acd1-3598b0bafa5d (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }, id: b7371311-29d9-4467-886d-e4e8f1bc545e, pool_id: ad348a98-27df-494e-af4b-560a8a421d64 (zpool) }] }, tx: ... } file = sled-storage/src/manager.rs:381
2192024-05-14T09:39:05.942ZINFOSledAgent (StorageManager): No ledger in /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2202024-05-14T09:39:05.942ZINFOSledAgent (StorageManager): No ledger in /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2212024-05-14T09:39:05.942ZINFOSledAgent (StorageManager): No previously-stored 'requested disks', creating new ledger file = sled-storage/src/manager.rs:681 request = omicron_physical_disks_ensure
2222024-05-14T09:39:05.942ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:339
2232024-05-14T09:39:05.942ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
2242024-05-14T09:39:05.942ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
2252024-05-14T09:39:05.942ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:112 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } path = /scratch/u2_0.vdev
2262024-05-14T09:39:06.101ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_0.vdev file = sled-hardware/src/disk.rs:304
2272024-05-14T09:39:06.101ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool)
2282024-05-14T09:39:06.399ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 511.254745ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:393 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
2292024-05-14T09:39:06.810ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/5f617488-2ef8-4cf1-81f7-2b249f5b34c0): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
2302024-05-14T09:39:07.002ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 1.568255936s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:393 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
2312024-05-14T09:39:07.236ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }
2322024-05-14T09:39:07.474ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
2332024-05-14T09:39:07.474ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
2342024-05-14T09:39:07.474ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
2352024-05-14T09:39:07.474ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
2362024-05-14T09:39:07.474ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2372024-05-14T09:39:07.474ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2382024-05-14T09:39:07.474ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
2392024-05-14T09:39:07.689ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/5f617488-2ef8-4cf1-81f7-2b249f5b34c0): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
2402024-05-14T09:39:07.881ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_switch
2412024-05-14T09:39:08.401ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/5f617488-2ef8-4cf1-81f7-2b249f5b34c0): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
2422024-05-14T09:39:08.623ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2432024-05-14T09:39:08.845ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 40.41434ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2442024-05-14T09:39:08.847ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 4.404201184s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:393 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
2452024-05-14T09:39:08.853ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_c919f4ac-100e-4c1e-bd24-76220f4848fc/crypt/zone file = sled-storage/src/dataset.rs:395
2462024-05-14T09:39:09.531ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 120.201514ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2472024-05-14T09:39:09.766ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/5f617488-2ef8-4cf1-81f7-2b249f5b34c0): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
2482024-05-14T09:39:09.766ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/5f617488-2ef8-4cf1-81f7-2b249f5b34c0): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
2492024-05-14T09:39:09.982ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 281.145481ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2502024-05-14T09:39:10.649ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 318.439252ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2512024-05-14T09:39:10.851ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/5f617488-2ef8-4cf1-81f7-2b249f5b34c0): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
2522024-05-14T09:39:11.464ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 610.081334ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2532024-05-14T09:39:11.890ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/5f617488-2ef8-4cf1-81f7-2b249f5b34c0): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
2542024-05-14T09:39:11.891ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/5f617488-2ef8-4cf1-81f7-2b249f5b34c0): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
2552024-05-14T09:39:11.908ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_c919f4ac-100e-4c1e-bd24-76220f4848fc file = sled-storage/src/dataset.rs:480
2562024-05-14T09:39:12.289ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
2572024-05-14T09:39:12.290ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
2582024-05-14T09:39:12.290ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
2592024-05-14T09:39:12.290ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
2602024-05-14T09:39:12.290ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:112 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } path = /scratch/u2_1.vdev
2612024-05-14T09:39:12.495ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 850.330886ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2622024-05-14T09:39:12.505ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_1.vdev file = sled-hardware/src/disk.rs:304
2632024-05-14T09:39:12.505ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool)
2642024-05-14T09:39:12.903ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/5f617488-2ef8-4cf1-81f7-2b249f5b34c0): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
2652024-05-14T09:39:14.085ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/5f617488-2ef8-4cf1-81f7-2b249f5b34c0): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
2662024-05-14T09:39:14.085ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 9.172892393s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:393 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
2672024-05-14T09:39:14.085ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/5f617488-2ef8-4cf1-81f7-2b249f5b34c0): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
2682024-05-14T09:39:14.309ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }
2692024-05-14T09:39:14.784ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
2702024-05-14T09:39:14.784ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
2712024-05-14T09:39:14.785ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
2722024-05-14T09:39:14.785ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
2732024-05-14T09:39:14.785ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2742024-05-14T09:39:14.785ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2752024-05-14T09:39:14.785ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
2762024-05-14T09:39:15.526ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/5f617488-2ef8-4cf1-81f7-2b249f5b34c0): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
2772024-05-14T09:39:15.527ZINFOSledAgent (ServiceManager): Ensuring tfportrear0_0/ll exists in zone file = sled-agent/src/services.rs:2303
2782024-05-14T09:39:15.971ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2792024-05-14T09:39:16.182ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/5f617488-2ef8-4cf1-81f7-2b249f5b34c0): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
2802024-05-14T09:39:16.187ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_9d13b91d-5969-417a-bfea-ad89852618b9/crypt/zone file = sled-storage/src/dataset.rs:395
2812024-05-14T09:39:16.374ZINFOSledAgent (VmmReservoirManager): Finished setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:279 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
2822024-05-14T09:39:16.500ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_9d13b91d-5969-417a-bfea-ad89852618b9 file = sled-storage/src/dataset.rs:480
2832024-05-14T09:39:16.508ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
2842024-05-14T09:39:16.508ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
2852024-05-14T09:39:16.508ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
2862024-05-14T09:39:16.508ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
2872024-05-14T09:39:16.508ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:112 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } path = /scratch/u2_2.vdev
2882024-05-14T09:39:16.519ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_2.vdev file = sled-hardware/src/disk.rs:304
2892024-05-14T09:39:16.519ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 39005e7e-189e-49db-8463-a206621e7a54 (zpool)
2902024-05-14T09:39:16.612ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }
2912024-05-14T09:39:16.619ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
2922024-05-14T09:39:16.620ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
2932024-05-14T09:39:16.620ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
2942024-05-14T09:39:16.620ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
2952024-05-14T09:39:16.620ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
2962024-05-14T09:39:16.620ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
2972024-05-14T09:39:16.620ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
2982024-05-14T09:39:16.623ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/5f617488-2ef8-4cf1-81f7-2b249f5b34c0): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
2992024-05-14T09:39:16.656ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3002024-05-14T09:39:16.664ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_39005e7e-189e-49db-8463-a206621e7a54/crypt/zone file = sled-storage/src/dataset.rs:395
3012024-05-14T09:39:16.789ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_39005e7e-189e-49db-8463-a206621e7a54 file = sled-storage/src/dataset.rs:480
3022024-05-14T09:39:16.797ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
3032024-05-14T09:39:16.797ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
3042024-05-14T09:39:16.797ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
3052024-05-14T09:39:16.797ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3062024-05-14T09:39:16.797ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:112 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } path = /scratch/u2_3.vdev
3072024-05-14T09:39:16.807ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_3.vdev file = sled-hardware/src/disk.rs:304
3082024-05-14T09:39:16.807ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = a7771067-edf5-47c9-a972-69320dc9ee62 (zpool)
3092024-05-14T09:39:16.899ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }
3102024-05-14T09:39:16.907ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
3112024-05-14T09:39:16.907ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
3122024-05-14T09:39:16.907ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
3132024-05-14T09:39:16.907ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
3142024-05-14T09:39:16.907ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3152024-05-14T09:39:16.907ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3162024-05-14T09:39:16.907ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3172024-05-14T09:39:16.940ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3182024-05-14T09:39:16.948ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_a7771067-edf5-47c9-a972-69320dc9ee62/crypt/zone file = sled-storage/src/dataset.rs:395
3192024-05-14T09:39:17.077ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_a7771067-edf5-47c9-a972-69320dc9ee62 file = sled-storage/src/dataset.rs:480
3202024-05-14T09:39:17.086ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
3212024-05-14T09:39:17.086ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
3222024-05-14T09:39:17.086ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
3232024-05-14T09:39:17.086ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3242024-05-14T09:39:17.086ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:112 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } path = /scratch/u2_4.vdev
3252024-05-14T09:39:17.096ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_4.vdev file = sled-hardware/src/disk.rs:304
3262024-05-14T09:39:17.096ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool)
3272024-05-14T09:39:17.192ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }
3282024-05-14T09:39:17.201ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
3292024-05-14T09:39:17.201ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
3302024-05-14T09:39:17.201ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
3312024-05-14T09:39:17.201ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
3322024-05-14T09:39:17.201ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3332024-05-14T09:39:17.201ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3342024-05-14T09:39:17.201ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3352024-05-14T09:39:17.237ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3362024-05-14T09:39:17.246ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_096e5012-bddf-482f-8ad5-65f3b0930676/crypt/zone file = sled-storage/src/dataset.rs:395
3372024-05-14T09:39:17.372ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_096e5012-bddf-482f-8ad5-65f3b0930676 file = sled-storage/src/dataset.rs:480
3382024-05-14T09:39:17.380ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
3392024-05-14T09:39:17.380ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
3402024-05-14T09:39:17.380ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
3412024-05-14T09:39:17.380ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3422024-05-14T09:39:17.380ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:112 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } path = /scratch/u2_5.vdev
3432024-05-14T09:39:17.390ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_5.vdev file = sled-hardware/src/disk.rs:304
3442024-05-14T09:39:17.390ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = ae2f89f7-35c8-4c05-940c-cdee6d8f8a12 (zpool)
3452024-05-14T09:39:17.484ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: ae2f89f7-35c8-4c05-940c-cdee6d8f8a12 (zpool), kind: External }
3462024-05-14T09:39:17.492ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
3472024-05-14T09:39:17.492ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
3482024-05-14T09:39:17.492ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
3492024-05-14T09:39:17.492ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
3502024-05-14T09:39:17.492ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3512024-05-14T09:39:17.492ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3522024-05-14T09:39:17.492ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3532024-05-14T09:39:17.527ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3542024-05-14T09:39:17.535ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_ae2f89f7-35c8-4c05-940c-cdee6d8f8a12/crypt/zone file = sled-storage/src/dataset.rs:395
3552024-05-14T09:39:17.625ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/5f617488-2ef8-4cf1-81f7-2b249f5b34c0): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
3562024-05-14T09:39:17.664ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_ae2f89f7-35c8-4c05-940c-cdee6d8f8a12 file = sled-storage/src/dataset.rs:480
3572024-05-14T09:39:17.672ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
3582024-05-14T09:39:17.672ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
3592024-05-14T09:39:17.672ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
3602024-05-14T09:39:17.672ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3612024-05-14T09:39:17.672ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:112 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } path = /scratch/u2_6.vdev
3622024-05-14T09:39:17.682ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_6.vdev file = sled-hardware/src/disk.rs:304
3632024-05-14T09:39:17.682ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 42b2209d-f16a-4fca-a84d-54ecc9f13877 (zpool)
3642024-05-14T09:39:17.775ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 42b2209d-f16a-4fca-a84d-54ecc9f13877 (zpool), kind: External }
3652024-05-14T09:39:17.782ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
3662024-05-14T09:39:17.782ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
3672024-05-14T09:39:17.782ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
3682024-05-14T09:39:17.782ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
3692024-05-14T09:39:17.782ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3702024-05-14T09:39:17.783ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3712024-05-14T09:39:17.783ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3722024-05-14T09:39:17.815ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3732024-05-14T09:39:17.822ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_42b2209d-f16a-4fca-a84d-54ecc9f13877/crypt/zone file = sled-storage/src/dataset.rs:395
3742024-05-14T09:39:17.941ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_42b2209d-f16a-4fca-a84d-54ecc9f13877 file = sled-storage/src/dataset.rs:480
3752024-05-14T09:39:17.949ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
3762024-05-14T09:39:17.949ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
3772024-05-14T09:39:17.949ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
3782024-05-14T09:39:17.949ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3792024-05-14T09:39:17.949ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:112 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } path = /scratch/u2_7.vdev
3802024-05-14T09:39:17.960ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_7.vdev file = sled-hardware/src/disk.rs:304
3812024-05-14T09:39:17.960ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 95ce52af-b115-45d4-acd1-3598b0bafa5d (zpool)
3822024-05-14T09:39:18.057ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 95ce52af-b115-45d4-acd1-3598b0bafa5d (zpool), kind: External }
3832024-05-14T09:39:18.065ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
3842024-05-14T09:39:18.065ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
3852024-05-14T09:39:18.065ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
3862024-05-14T09:39:18.065ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
3872024-05-14T09:39:18.065ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
3882024-05-14T09:39:18.065ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
3892024-05-14T09:39:18.065ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3902024-05-14T09:39:18.086ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/5f617488-2ef8-4cf1-81f7-2b249f5b34c0): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
3912024-05-14T09:39:18.102ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3922024-05-14T09:39:18.109ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_95ce52af-b115-45d4-acd1-3598b0bafa5d/crypt/zone file = sled-storage/src/dataset.rs:395
3932024-05-14T09:39:18.245ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_95ce52af-b115-45d4-acd1-3598b0bafa5d file = sled-storage/src/dataset.rs:480
3942024-05-14T09:39:18.253ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
3952024-05-14T09:39:18.253ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
3962024-05-14T09:39:18.253ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
3972024-05-14T09:39:18.253ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3982024-05-14T09:39:18.253ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:112 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } path = /scratch/u2_8.vdev
3992024-05-14T09:39:18.264ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_8.vdev file = sled-hardware/src/disk.rs:304
4002024-05-14T09:39:18.264ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = ad348a98-27df-494e-af4b-560a8a421d64 (zpool)
4012024-05-14T09:39:18.366ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: ad348a98-27df-494e-af4b-560a8a421d64 (zpool), kind: External }
4022024-05-14T09:39:18.373ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
4032024-05-14T09:39:18.373ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
4042024-05-14T09:39:18.373ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
4052024-05-14T09:39:18.373ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
4062024-05-14T09:39:18.373ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4072024-05-14T09:39:18.373ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4082024-05-14T09:39:18.373ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
4092024-05-14T09:39:18.408ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4102024-05-14T09:39:18.416ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_ad348a98-27df-494e-af4b-560a8a421d64/crypt/zone file = sled-storage/src/dataset.rs:395
4112024-05-14T09:39:18.544ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_ad348a98-27df-494e-af4b-560a8a421d64 file = sled-storage/src/dataset.rs:480
4122024-05-14T09:39:18.552ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
4132024-05-14T09:39:18.552ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
4142024-05-14T09:39:18.552ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4152024-05-14T09:39:18.552ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 112d7827-193f-456c-94a5-19163d29475e (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 }, zpool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: e21e7560-2c98-46a4-bdda-f772e50ec2a3 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 }, zpool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 }, zpool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 }, zpool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, variant: U2, slot: 1030 }, zpool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, variant: U2, slot: 1031 }, zpool_name: ZpoolName { id: ae2f89f7-35c8-4c05-940c-cdee6d8f8a12 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }, variant: U2, slot: 1032 }, zpool_name: ZpoolName { id: 42b2209d-f16a-4fca-a84d-54ecc9f13877 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }, variant: U2, slot: 1033 }, zpool_name: ZpoolName { id: 95ce52af-b115-45d4-acd1-3598b0bafa5d (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }, variant: U2, slot: 1034 }, zpool_name: ZpoolName { id: ad348a98-27df-494e-af4b-560a8a421d64 (zpool), kind: External } }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
4162024-05-14T09:39:18.552ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4172024-05-14T09:39:18.553ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 12611121 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:55076 req_id = 13dc1362-9678-457b-b160-6a88f80da4fc response_code = 200 uri = /omicron-physical-disks
4182024-05-14T09:39:18.583ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(2), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
4192024-05-14T09:39:18.583ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:36652
4202024-05-14T09:39:18.584ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
4212024-05-14T09:39:18.584ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
4222024-05-14T09:39:18.628ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/5f617488-2ef8-4cf1-81f7-2b249f5b34c0): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
4232024-05-14T09:39:18.636ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
4242024-05-14T09:39:18.636ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4252024-05-14T09:39:18.687ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
4262024-05-14T09:39:18.687ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4272024-05-14T09:39:18.738ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4282024-05-14T09:39:18.738ZINFOSledAgent (ServiceManager): No ledger in /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:188
4292024-05-14T09:39:18.738ZINFOSledAgent (ServiceManager): No ledger in /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:188
4302024-05-14T09:39:18.738ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4312024-05-14T09:39:18.786ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4322024-05-14T09:39:18.786ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4332024-05-14T09:39:18.786ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4342024-05-14T09:39:18.841ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_12f3cc35-fd2a-411b-bc60-c7832113465e file = illumos-utils/src/zone.rs:315
4352024-05-14T09:39:18.845ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_baf23778-f6a3-4d4f-931a-1389406eba07 file = illumos-utils/src/zone.rs:315
4362024-05-14T09:39:18.849ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_063ecfd8-b3e0-46f6-98ae-94ccfb48c08c file = illumos-utils/src/zone.rs:315
4372024-05-14T09:39:18.860ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_12f3cc35-fd2a-411b-bc60-c7832113465e file = illumos-utils/src/zone.rs:354
4382024-05-14T09:39:18.865ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_baf23778-f6a3-4d4f-931a-1389406eba07 file = illumos-utils/src/zone.rs:354
4392024-05-14T09:39:18.869ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_063ecfd8-b3e0-46f6-98ae-94ccfb48c08c file = illumos-utils/src/zone.rs:354
4402024-05-14T09:39:19.630ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/5f617488-2ef8-4cf1-81f7-2b249f5b34c0): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
4412024-05-14T09:39:20.386ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_063ecfd8-b3e0-46f6-98ae-94ccfb48c08c: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService3"/> <propval type="astring" name="gateway" value="fd00:1122:3344:2::2"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:2::1"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:2::1]:5353"/> <propval type="astring" name="dns_address" value="[fd00:1122:3344:2::1]:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4422024-05-14T09:39:20.386ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:87 prefix = Ipv6Prefix { addr: fd00:1122:3344:2::, len: 64 }
4432024-05-14T09:39:20.863ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_baf23778-f6a3-4d4f-931a-1389406eba07: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService2"/> <propval type="astring" name="gateway" value="fd00:1122:3344:3::2"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:3::1"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:3::1]:5353"/> <propval type="astring" name="dns_address" value="[fd00:1122:3344:3::1]:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4442024-05-14T09:39:20.863ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_internal_dns_063ecfd8-b3e0-46f6-98ae-94ccfb48c08c
4452024-05-14T09:39:20.867ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:87 prefix = Ipv6Prefix { addr: fd00:1122:3344:3::, len: 64 }
4462024-05-14T09:39:21.349ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_12f3cc35-fd2a-411b-bc60-c7832113465e: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService1"/> <propval type="astring" name="gateway" value="fd00:1122:3344:1::2"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:1::1"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:1::1]:5353"/> <propval type="astring" name="dns_address" value="[fd00:1122:3344:1::1]:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4472024-05-14T09:39:21.349ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_internal_dns_baf23778-f6a3-4d4f-931a-1389406eba07
4482024-05-14T09:39:21.366ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_063ecfd8-b3e0-46f6-98ae-94ccfb48c08c") failed: Property not found. retry in 63.267647ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_063ecfd8-b3e0-46f6-98ae-94ccfb48c08c
4492024-05-14T09:39:21.366ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:87 prefix = Ipv6Prefix { addr: fd00:1122:3344:1::, len: 64 }
4502024-05-14T09:39:21.366ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_internal_dns_12f3cc35-fd2a-411b-bc60-c7832113465e
4512024-05-14T09:39:21.439ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_063ecfd8-b3e0-46f6-98ae-94ccfb48c08c") failed: Property not found. retry in 136.481809ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_063ecfd8-b3e0-46f6-98ae-94ccfb48c08c
4522024-05-14T09:39:21.477ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_baf23778-f6a3-4d4f-931a-1389406eba07") failed: Property not found. retry in 58.507704ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_baf23778-f6a3-4d4f-931a-1389406eba07
4532024-05-14T09:39:21.485ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_12f3cc35-fd2a-411b-bc60-c7832113465e") failed: Property not found. retry in 32.318312ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_12f3cc35-fd2a-411b-bc60-c7832113465e
4542024-05-14T09:39:21.526ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_12f3cc35-fd2a-411b-bc60-c7832113465e") failed: Property not found. retry in 97.637734ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_12f3cc35-fd2a-411b-bc60-c7832113465e
4552024-05-14T09:39:21.550ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_baf23778-f6a3-4d4f-931a-1389406eba07") failed: Property not found. retry in 114.550682ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_baf23778-f6a3-4d4f-931a-1389406eba07
4562024-05-14T09:39:21.625ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_063ecfd8-b3e0-46f6-98ae-94ccfb48c08c") failed: Property not found. retry in 125.198734ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_063ecfd8-b3e0-46f6-98ae-94ccfb48c08c
4572024-05-14T09:39:21.646ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_12f3cc35-fd2a-411b-bc60-c7832113465e") failed: Property not found. retry in 195.593066ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_12f3cc35-fd2a-411b-bc60-c7832113465e
4582024-05-14T09:39:21.699ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_baf23778-f6a3-4d4f-931a-1389406eba07") failed: Property not found. retry in 142.93638ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_baf23778-f6a3-4d4f-931a-1389406eba07
4592024-05-14T09:39:21.764ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_063ecfd8-b3e0-46f6-98ae-94ccfb48c08c") failed: Property not found. retry in 426.597544ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_063ecfd8-b3e0-46f6-98ae-94ccfb48c08c
4602024-05-14T09:39:21.857ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_baf23778-f6a3-4d4f-931a-1389406eba07") failed: Property not found. retry in 265.438314ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_baf23778-f6a3-4d4f-931a-1389406eba07
4612024-05-14T09:39:21.873ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_12f3cc35-fd2a-411b-bc60-c7832113465e") failed: Property not found. retry in 501.804692ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_12f3cc35-fd2a-411b-bc60-c7832113465e
4622024-05-14T09:39:22.138ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_baf23778-f6a3-4d4f-931a-1389406eba07") failed: Property not found. retry in 641.34092ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_baf23778-f6a3-4d4f-931a-1389406eba07
4632024-05-14T09:39:22.204ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_063ecfd8-b3e0-46f6-98ae-94ccfb48c08c") failed: Property not found. retry in 513.18259ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_063ecfd8-b3e0-46f6-98ae-94ccfb48c08c
4642024-05-14T09:39:22.383ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_12f3cc35-fd2a-411b-bc60-c7832113465e") failed: Property not found. retry in 488.920041ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_12f3cc35-fd2a-411b-bc60-c7832113465e
4652024-05-14T09:39:22.727ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_063ecfd8-b3e0-46f6-98ae-94ccfb48c08c") failed: Property not found. retry in 562.065687ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_063ecfd8-b3e0-46f6-98ae-94ccfb48c08c
4662024-05-14T09:39:22.789ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_baf23778-f6a3-4d4f-931a-1389406eba07") failed: Property not found. retry in 566.551012ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_baf23778-f6a3-4d4f-931a-1389406eba07
4672024-05-14T09:39:22.881ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_12f3cc35-fd2a-411b-bc60-c7832113465e") failed: Property not found. retry in 823.968007ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_12f3cc35-fd2a-411b-bc60-c7832113465e
4682024-05-14T09:39:23.299ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_063ecfd8-b3e0-46f6-98ae-94ccfb48c08c") failed: Property not found. retry in 738.075373ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_063ecfd8-b3e0-46f6-98ae-94ccfb48c08c
4692024-05-14T09:39:23.366ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_baf23778-f6a3-4d4f-931a-1389406eba07") failed: Property not found. retry in 912.349393ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_baf23778-f6a3-4d4f-931a-1389406eba07
4702024-05-14T09:39:23.623ZINFOSledAgent (ServiceManager): Ensuring bootstrap address fdb0:18c0:4d0d:a02a::2 exists in switch zone file = sled-agent/src/services.rs:2320
4712024-05-14T09:39:23.623ZINFOSledAgent (ServiceManager): Adding bootstrap address file = illumos-utils/src/running_zone.rs:626 zone = oxz_switch
4722024-05-14T09:39:23.714ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_12f3cc35-fd2a-411b-bc60-c7832113465e") failed: Property not found. retry in 781.73021ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_12f3cc35-fd2a-411b-bc60-c7832113465e
4732024-05-14T09:39:24.046ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_063ecfd8-b3e0-46f6-98ae-94ccfb48c08c") failed: Property not found. retry in 1.483623306s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_063ecfd8-b3e0-46f6-98ae-94ccfb48c08c
4742024-05-14T09:39:24.173ZINFOSledAgent (ServiceManager): Forwarding bootstrap traffic via oxBootstrap0 to fe80::8:20ff:fe7e:e4da file = sled-agent/src/services.rs:2327
4752024-05-14T09:39:24.179ZINFOSledAgent (ServiceManager): Zone not using gateway (even though underlay is up) file = sled-agent/src/services.rs:2388
4762024-05-14T09:39:24.286ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_baf23778-f6a3-4d4f-931a-1389406eba07") failed: Property not found. retry in 1.024174813s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_baf23778-f6a3-4d4f-931a-1389406eba07
4772024-05-14T09:39:24.303ZINFOSledAgent (ServiceManager): Setting up dendrite service file = sled-agent/src/services.rs:2573
4782024-05-14T09:39:24.505ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_12f3cc35-fd2a-411b-bc60-c7832113465e") failed: Property not found. retry in 764.926745ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_12f3cc35-fd2a-411b-bc60-c7832113465e
4792024-05-14T09:39:24.559ZINFOSledAgent (ServiceManager): Setting up lldpd service file = sled-agent/src/services.rs:2737
4802024-05-14T09:39:24.763ZINFOSledAgent (ServiceManager): Setting up MGS service file = sled-agent/src/services.rs:2452
4812024-05-14T09:39:25.073ZINFOSledAgent (ServiceManager): Setting up wicketd service file = sled-agent/src/services.rs:2487
4822024-05-14T09:39:25.280ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_12f3cc35-fd2a-411b-bc60-c7832113465e") failed: Property not found. retry in 1.149097928s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_12f3cc35-fd2a-411b-bc60-c7832113465e
4832024-05-14T09:39:25.284ZINFOSledAgent (ServiceManager): Setting up mgd service file = sled-agent/src/services.rs:2787
4842024-05-14T09:39:25.319ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_baf23778-f6a3-4d4f-931a-1389406eba07") failed: Property not found. retry in 1.264373424s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_baf23778-f6a3-4d4f-931a-1389406eba07
4852024-05-14T09:39:25.472ZINFOSledAgent (ServiceManager): Setting up mg-ddm service file = sled-agent/src/services.rs:2814
4862024-05-14T09:39:25.538ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_063ecfd8-b3e0-46f6-98ae-94ccfb48c08c") failed: Property not found. retry in 1.221625926s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_063ecfd8-b3e0-46f6-98ae-94ccfb48c08c
4872024-05-14T09:39:25.686ZINFOSledAgent (ServiceManager): Setting up tfport service file = sled-agent/src/services.rs:2681
4882024-05-14T09:39:25.878ZINFOSledAgent (ServiceManager): Setting up Simulated SP service file = sled-agent/src/services.rs:2481
4892024-05-14T09:39:25.889ZINFOSledAgent (ServiceManager): Re-enabling running switch zone (new address) file = sled-agent/src/services.rs:3873 new = [fd00:1122:3344:101::2, ::1] old = [::1]
4902024-05-14T09:39:25.889ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists file = sled-agent/src/services.rs:3889
4912024-05-14T09:39:25.889ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::2, prefix: 64 })) file = illumos-utils/src/running_zone.rs:609 zone = oxz_switch
4922024-05-14T09:39:26.446ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists - OK file = sled-agent/src/services.rs:3897
4932024-05-14T09:39:26.452ZINFOSledAgent (ServiceManager): configuring dendrite service file = sled-agent/src/services.rs:3953
4942024-05-14T09:39:26.614ZINFOSledAgent (ServiceManager): configuring lldp service file = sled-agent/src/services.rs:4015
4952024-05-14T09:39:26.728ZINFOSledAgent (ServiceManager): configuring wicketd file = sled-agent/src/services.rs:3996 rack_subnet = fd00:1122:3344::
4962024-05-14T09:39:26.753ZINFOSledAgent (ServiceManager): configuring mgd service file = sled-agent/src/services.rs:4043
4972024-05-14T09:39:26.782ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_internal_dns_12f3cc35-fd2a-411b-bc60-c7832113465e
4982024-05-14T09:39:26.782ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_internal_dns_baf23778-f6a3-4d4f-931a-1389406eba07
4992024-05-14T09:39:26.782ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_internal_dns_063ecfd8-b3e0-46f6-98ae-94ccfb48c08c
5002024-05-14T09:39:26.782ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5012024-05-14T09:39:26.783ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5022024-05-14T09:39:26.783ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 8199517 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:36652 req_id = 2b1f0bdd-e4c1-46e4-b279-235cdffd3c71 response_code = 204 uri = /omicron-zones
5032024-05-14T09:39:26.785ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:1::1]:5353 file = sled-agent/src/rack_setup/service.rs:548
5042024-05-14T09:39:26.861ZINFOSledAgent (ServiceManager): configuring mg-ddm service file = sled-agent/src/services.rs:4070
5052024-05-14T09:39:26.985ZINFOSledAgent (ServiceManager): Determining physical location of our switch zone at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:365
5062024-05-14T09:39:27.016ZINFOSledAgent (ServiceManager): Initializing 1 Uplinks on Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:413
5072024-05-14T09:39:27.048ZINFOSledAgent (ServiceManager): Building Port Configuration file = sled-agent/src/bootstrap/early_networking.rs:651
5082024-05-14T09:39:27.048ZINFOSledAgent (ServiceManager): Checking dendrite uptime file = sled-agent/src/bootstrap/early_networking.rs:689
5092024-05-14T09:39:27.048ZINFOSledAgent (ServiceManager): Dendrite online file = sled-agent/src/bootstrap/early_networking.rs:692 uptime = 3
5102024-05-14T09:39:27.048ZINFOSledAgent (ServiceManager): Configuring default uplink on switch config = PortSettings {\n links: {\n "0": LinkSettings {\n addrs: [\n 10.151.2.174,\n ],\n params: LinkCreate {\n autoneg: false,\n fec: None,\n kr: false,\n lane: Some(\n LinkId(\n 0,\n ),\n ),\n speed: Speed40G,\n },\n },\n },\n} file = sled-agent/src/bootstrap/early_networking.rs:435
5112024-05-14T09:39:27.122ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:381
5122024-05-14T09:39:27.122ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }
5132024-05-14T09:39:27.122ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }
5142024-05-14T09:39:27.122ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }
5152024-05-14T09:39:27.122ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }
5162024-05-14T09:39:27.122ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }
5172024-05-14T09:39:27.122ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }
5182024-05-14T09:39:27.122ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }
5192024-05-14T09:39:27.122ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }
5202024-05-14T09:39:27.122ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }
5212024-05-14T09:39:27.122ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }
5222024-05-14T09:39:27.122ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }
5232024-05-14T09:39:27.122ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }
5242024-05-14T09:39:27.122ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }
5252024-05-14T09:39:27.122ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }
5262024-05-14T09:39:27.122ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }
5272024-05-14T09:39:27.122ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }
5282024-05-14T09:39:27.122ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }
5292024-05-14T09:39:27.122ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }
5302024-05-14T09:39:27.122ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }
5312024-05-14T09:39:27.122ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }
5322024-05-14T09:39:27.122ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }
5332024-05-14T09:39:27.122ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }
5342024-05-14T09:39:27.225ZWARNSledAgent (RSS): failed to write DNS configuration (will retry in 56.875444ms) dns_config_addr = [fd00:1122:3344:1::1]:5353 error_message = Communication Error: error sending request for url (http://[fd00:1122:3344:1::1]:5353/config): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:568
5352024-05-14T09:39:27.300ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:2::1]:5353 file = sled-agent/src/rack_setup/service.rs:548
5362024-05-14T09:39:27.347ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:3::1]:5353 file = sled-agent/src/rack_setup/service.rs:548
5372024-05-14T09:39:27.391ZINFOSledAgent (RSS): Configured all DNS servers file = sled-agent/src/rack_setup/service.rs:584
5382024-05-14T09:39:27.391ZINFOSledAgent (RSS): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:259
5392024-05-14T09:39:27.423ZINFOSledAgent (RSS): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:276
5402024-05-14T09:39:27.424ZINFOSledAgent (RSS): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:301
5412024-05-14T09:39:27.424ZINFOSledAgent (RSS): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:193
5422024-05-14T09:39:27.452ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(3), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
5432024-05-14T09:39:27.452ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:49117
5442024-05-14T09:39:27.452ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
5452024-05-14T09:39:27.452ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
5462024-05-14T09:39:27.473ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
5472024-05-14T09:39:27.473ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5482024-05-14T09:39:27.493ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
5492024-05-14T09:39:27.493ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5502024-05-14T09:39:27.512ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
5512024-05-14T09:39:27.512ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
5522024-05-14T09:39:27.513ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
5532024-05-14T09:39:27.513ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
5542024-05-14T09:39:27.525ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:259
5552024-05-14T09:39:27.556ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:276
5562024-05-14T09:39:27.557ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:301
5572024-05-14T09:39:27.557ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:193
5582024-05-14T09:39:27.601ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:394 port = Port { inner: PortInner { name: "opte0", ip: 172.30.3.5, mac: MacAddr6([168, 64, 37, 255, 240, 111]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.3.1 }, vnic: "vopte0" } }
5592024-05-14T09:39:27.601ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1290 zone_type = ntp
5602024-05-14T09:39:27.601ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.2.169, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
5612024-05-14T09:39:27.601ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.2.169, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
5622024-05-14T09:39:27.602ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.2.169, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
5632024-05-14T09:39:27.604ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
5642024-05-14T09:39:27.628ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_ntp_2d5804f9-5e88-4859-b528-dca669659405 file = illumos-utils/src/zone.rs:315
5652024-05-14T09:39:27.646ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_ntp_2d5804f9-5e88-4859-b528-dca669659405 file = illumos-utils/src/zone.rs:354
5662024-05-14T09:39:28.297ZINFOSledAgent (ServiceManager): Profile for oxz_ntp_2d5804f9-5e88-4859-b528-dca669659405: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService4"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::1b"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/chrony-setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="allow" value="fd00:1122:3344:100::/56"/> <propval type="boolean" name="boundary" value="true"/> <propval type="astring" name="server" value="0.pool.ntp.org"/> </property_group> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="1.1.1.1" /> <value_node value="9.9.9.9" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/ntp"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte0"/> <propval type="astring" name="gateway" value="172.30.3.1"/> <propval type="astring" name="ip" value="172.30.3.5"/> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
5672024-05-14T09:39:28.297ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_ntp_2d5804f9-5e88-4859-b528-dca669659405
5682024-05-14T09:39:28.447ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_2d5804f9-5e88-4859-b528-dca669659405") failed: Property not found. retry in 33.604941ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_2d5804f9-5e88-4859-b528-dca669659405
5692024-05-14T09:39:28.488ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_2d5804f9-5e88-4859-b528-dca669659405") failed: Property not found. retry in 110.177972ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_2d5804f9-5e88-4859-b528-dca669659405
5702024-05-14T09:39:28.628ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_2d5804f9-5e88-4859-b528-dca669659405") failed: Property not found. retry in 218.044964ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_2d5804f9-5e88-4859-b528-dca669659405
5712024-05-14T09:39:28.858ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_2d5804f9-5e88-4859-b528-dca669659405") failed: Property not found. retry in 401.603672ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_2d5804f9-5e88-4859-b528-dca669659405
5722024-05-14T09:39:29.268ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_2d5804f9-5e88-4859-b528-dca669659405") failed: Property not found. retry in 1.058527268s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_2d5804f9-5e88-4859-b528-dca669659405
5732024-05-14T09:39:30.336ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_2d5804f9-5e88-4859-b528-dca669659405") failed: Property not found. retry in 1.46509076s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_2d5804f9-5e88-4859-b528-dca669659405
5742024-05-14T09:39:30.411ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:34210
5752024-05-14T09:39:30.411ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 59 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:34210 req_id = b1840452-6b7d-4f1d-8778-f297864f0b3f response_code = 200 uri = /baseboard
5762024-05-14T09:39:31.811ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_2d5804f9-5e88-4859-b528-dca669659405") failed: Property not found. retry in 1.3029849s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_2d5804f9-5e88-4859-b528-dca669659405
5772024-05-14T09:39:33.123ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_2d5804f9-5e88-4859-b528-dca669659405") failed: Property not found. retry in 549.981005ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_2d5804f9-5e88-4859-b528-dca669659405
5782024-05-14T09:39:33.695ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_ntp_2d5804f9-5e88-4859-b528-dca669659405
5792024-05-14T09:39:33.695ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5802024-05-14T09:39:33.696ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5812024-05-14T09:39:33.696ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 6243642 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49117 req_id = a5c899b6-31b8-4786-b0cd-25718915ef1b response_code = 204 uri = /omicron-zones
5822024-05-14T09:39:33.698ZINFOSledAgent (RSS): Waiting for rack time synchronization file = sled-agent/src/rack_setup/service.rs:625
5832024-05-14T09:39:33.728ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:605
5842024-05-14T09:39:33.728ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:49560
5852024-05-14T09:39:33.738ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_2d5804f9-5e88-4859-b528-dca669659405': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3603
5862024-05-14T09:39:33.738ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 9903 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:49560 req_id = c6e84882-134c-406d-b8b0-d271c161ffdb response_code = 500 uri = /timesync
5872024-05-14T09:39:33.741ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:655
5882024-05-14T09:39:33.911ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:605
5892024-05-14T09:39:33.911ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:53449
5902024-05-14T09:39:37.920ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_2d5804f9-5e88-4859-b528-dca669659405': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3603
5912024-05-14T09:39:37.920ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 4008631 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:53449 req_id = 838908cb-2732-40f2-a97f-a994a83a6785 response_code = 500 uri = /timesync
5922024-05-14T09:39:37.923ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:655
5932024-05-14T09:39:38.023ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:605
5942024-05-14T09:39:38.023ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:36070
5952024-05-14T09:39:38.031ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_2d5804f9-5e88-4859-b528-dca669659405': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3603
5962024-05-14T09:39:38.032ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 8628 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:36070 req_id = 53c8a11b-98a3-4b0b-91f4-d29650c14bcf response_code = 500 uri = /timesync
5972024-05-14T09:39:38.033ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:655
5982024-05-14T09:39:38.150ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:605
5992024-05-14T09:39:38.150ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:52763
6002024-05-14T09:39:42.159ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_2d5804f9-5e88-4859-b528-dca669659405': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3603
6012024-05-14T09:39:42.159ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 4008847 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:52763 req_id = 6552caf7-80ce-4f42-b510-71119911c97e response_code = 500 uri = /timesync
6022024-05-14T09:39:42.161ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:655
6032024-05-14T09:39:42.305ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:605
6042024-05-14T09:39:42.305ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:61465
6052024-05-14T09:39:42.315ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 9401 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:61465 req_id = cd16ca93-f712-4b65-b699-6be7c676cb7e response_code = 200 uri = /timesync
6062024-05-14T09:39:42.317ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:633
6072024-05-14T09:39:42.317ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:655
6082024-05-14T09:39:42.624ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:605
6092024-05-14T09:39:42.624ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:64381
6102024-05-14T09:39:42.632ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 8030 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:64381 req_id = ffa7510d-1a0c-4e2e-b04a-b89c449c2bab response_code = 200 uri = /timesync
6112024-05-14T09:39:42.635ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:633
6122024-05-14T09:39:42.635ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:655
6132024-05-14T09:39:42.892ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:605
6142024-05-14T09:39:42.892ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:54951
6152024-05-14T09:39:42.902ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 9237 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:54951 req_id = b86bc6d1-8d2c-4028-b382-97fb76a519da response_code = 200 uri = /timesync
6162024-05-14T09:39:42.904ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:633
6172024-05-14T09:39:42.904ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:655
6182024-05-14T09:39:43.221ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:605
6192024-05-14T09:39:43.221ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:33410
6202024-05-14T09:39:43.229ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 7768 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:33410 req_id = b3f139a2-1e30-4741-9869-8070cae7ee90 response_code = 200 uri = /timesync
6212024-05-14T09:39:43.230ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:633
6222024-05-14T09:39:43.230ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:655
6232024-05-14T09:39:43.299ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 309.628126ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:426 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
6242024-05-14T09:39:43.792ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:605
6252024-05-14T09:39:43.792ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:49239
6262024-05-14T09:39:43.801ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 8248 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:49239 req_id = 04a2c1bf-642d-482c-a832-0b26ef176038 response_code = 200 uri = /timesync
6272024-05-14T09:39:43.802ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:633
6282024-05-14T09:39:43.802ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:655
6292024-05-14T09:39:44.092ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:605
6302024-05-14T09:39:44.092ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42874
6312024-05-14T09:39:44.100ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 7686 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:42874 req_id = 384d00f8-79d6-4880-a3eb-50b82b07c61f response_code = 200 uri = /timesync
6322024-05-14T09:39:44.102ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:633
6332024-05-14T09:39:44.102ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:655
6342024-05-14T09:39:44.476ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:605
6352024-05-14T09:39:44.476ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:44946
6362024-05-14T09:39:44.485ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 8871 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:44946 req_id = 455fa1f2-40d2-4509-a227-fa296c709350 response_code = 200 uri = /timesync
6372024-05-14T09:39:44.488ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:633
6382024-05-14T09:39:44.488ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:655
6392024-05-14T09:39:44.265ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:605
6402024-05-14T09:39:44.265ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:61123
6412024-05-14T09:39:44.268ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:801
6422024-05-14T09:39:44.289ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 23866 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:61123 req_id = a4a6f110-0e2a-4999-9702-8fb63ef48ff2 response_code = 200 uri = /timesync
6432024-05-14T09:39:44.290ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: true, ref_id: 396892370, ip_addr: 23.168.24.210, stratum: 3, ref_time: 1715679584.1683795, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:633
6442024-05-14T09:39:44.291ZINFOSledAgent (RSS): Finished setting up Internal DNS and NTP file = sled-agent/src/rack_setup/service.rs:1216
6452024-05-14T09:39:44.319ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
6462024-05-14T09:39:44.319ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:59710
6472024-05-14T09:39:44.319ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
6482024-05-14T09:39:44.319ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
6492024-05-14T09:39:44.339ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
6502024-05-14T09:39:44.339ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6512024-05-14T09:39:44.359ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
6522024-05-14T09:39:44.359ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6532024-05-14T09:39:44.378ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6542024-05-14T09:39:44.378ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6552024-05-14T09:39:44.431ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6562024-05-14T09:39:44.431ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6572024-05-14T09:39:44.485ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6582024-05-14T09:39:44.485ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6592024-05-14T09:39:44.538ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6602024-05-14T09:39:44.538ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6612024-05-14T09:39:44.590ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6622024-05-14T09:39:44.590ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6632024-05-14T09:39:44.641ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
6642024-05-14T09:39:44.641ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
6652024-05-14T09:39:44.642ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
6662024-05-14T09:39:44.642ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
6672024-05-14T09:39:44.674ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
6682024-05-14T09:39:44.675ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
6692024-05-14T09:39:44.675ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
6702024-05-14T09:39:44.675ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
6712024-05-14T09:39:44.675ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
6722024-05-14T09:39:44.675ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529", TimeNotSynchronized), ("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265", TimeNotSynchronized), ("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf", TimeNotSynchronized), ("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d", TimeNotSynchronized), ("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 355521 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59710 req_id = 337e361e-741f-4088-89b0-edb3707fc77f response_code = 500 uri = /omicron-zones
6732024-05-14T09:39:44.675ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "337e361e-741f-4088-89b0-edb3707fc77f", "content-length": "124", "date": "Tue, 14 May 2024 09:39:43 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "337e361e-741f-4088-89b0-edb3707fc77f" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 129.188ms sled_address = [fd00:1122:3344:101::1]:12345
6742024-05-14T09:39:44.805ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
6752024-05-14T09:39:44.806ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
6762024-05-14T09:39:44.806ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
6772024-05-14T09:39:44.825ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
6782024-05-14T09:39:44.825ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6792024-05-14T09:39:44.843ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
6802024-05-14T09:39:44.843ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6812024-05-14T09:39:44.861ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6822024-05-14T09:39:44.861ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6832024-05-14T09:39:44.879ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6842024-05-14T09:39:44.879ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6852024-05-14T09:39:44.896ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6862024-05-14T09:39:44.896ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6872024-05-14T09:39:44.914ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6882024-05-14T09:39:44.914ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6892024-05-14T09:39:44.933ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6902024-05-14T09:39:44.933ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6912024-05-14T09:39:44.951ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
6922024-05-14T09:39:44.951ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
6932024-05-14T09:39:44.951ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
6942024-05-14T09:39:44.951ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
6952024-05-14T09:39:44.984ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
6962024-05-14T09:39:44.984ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
6972024-05-14T09:39:44.984ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
6982024-05-14T09:39:44.984ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
6992024-05-14T09:39:44.984ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
7002024-05-14T09:39:44.984ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265", TimeNotSynchronized), ("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d", TimeNotSynchronized), ("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997", TimeNotSynchronized), ("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf", TimeNotSynchronized), ("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 178861 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59710 req_id = 383366e7-f2c4-4d98-acf5-3113ee71212d response_code = 500 uri = /omicron-zones
7012024-05-14T09:39:44.984ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "383366e7-f2c4-4d98-acf5-3113ee71212d", "content-length": "124", "date": "Tue, 14 May 2024 09:39:44 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "383366e7-f2c4-4d98-acf5-3113ee71212d" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 79.06442ms sled_address = [fd00:1122:3344:101::1]:12345
7022024-05-14T09:39:45.065ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
7032024-05-14T09:39:45.065ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
7042024-05-14T09:39:45.065ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
7052024-05-14T09:39:45.085ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
7062024-05-14T09:39:45.085ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7072024-05-14T09:39:45.102ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
7082024-05-14T09:39:45.102ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7092024-05-14T09:39:45.117ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7102024-05-14T09:39:45.117ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7112024-05-14T09:39:45.137ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7122024-05-14T09:39:45.137ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7132024-05-14T09:39:45.157ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7142024-05-14T09:39:45.157ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7152024-05-14T09:39:45.174ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7162024-05-14T09:39:45.174ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7172024-05-14T09:39:45.194ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7182024-05-14T09:39:45.194ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7192024-05-14T09:39:45.209ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
7202024-05-14T09:39:45.209ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
7212024-05-14T09:39:45.209ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
7222024-05-14T09:39:45.210ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
7232024-05-14T09:39:45.239ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
7242024-05-14T09:39:45.239ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
7252024-05-14T09:39:45.239ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
7262024-05-14T09:39:45.239ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
7272024-05-14T09:39:45.239ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
7282024-05-14T09:39:45.239ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d", TimeNotSynchronized), ("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997", TimeNotSynchronized), ("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265", TimeNotSynchronized), ("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529", TimeNotSynchronized), ("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 174637 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59710 req_id = a6e957b4-2421-4bff-82a3-b36e8ae9fd5f response_code = 500 uri = /omicron-zones
7292024-05-14T09:39:45.239ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "a6e957b4-2421-4bff-82a3-b36e8ae9fd5f", "content-length": "124", "date": "Tue, 14 May 2024 09:39:44 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "a6e957b4-2421-4bff-82a3-b36e8ae9fd5f" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 193.955876ms sled_address = [fd00:1122:3344:101::1]:12345
7302024-05-14T09:39:45.434ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
7312024-05-14T09:39:45.434ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
7322024-05-14T09:39:45.434ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
7332024-05-14T09:39:45.456ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
7342024-05-14T09:39:45.456ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7352024-05-14T09:39:45.473ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
7362024-05-14T09:39:45.473ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7372024-05-14T09:39:45.491ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7382024-05-14T09:39:45.491ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7392024-05-14T09:39:45.511ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7402024-05-14T09:39:45.511ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7412024-05-14T09:39:45.529ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7422024-05-14T09:39:45.529ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7432024-05-14T09:39:45.547ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7442024-05-14T09:39:45.547ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7452024-05-14T09:39:45.566ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7462024-05-14T09:39:45.566ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7472024-05-14T09:39:45.586ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
7482024-05-14T09:39:45.586ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
7492024-05-14T09:39:45.587ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
7502024-05-14T09:39:45.587ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
7512024-05-14T09:39:45.622ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
7522024-05-14T09:39:45.622ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
7532024-05-14T09:39:45.622ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
7542024-05-14T09:39:45.622ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
7552024-05-14T09:39:45.622ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
7562024-05-14T09:39:45.622ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265", TimeNotSynchronized), ("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529", TimeNotSynchronized), ("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d", TimeNotSynchronized), ("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf", TimeNotSynchronized), ("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 187796 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59710 req_id = cfb226ee-3cb4-4e20-8e39-2b76e9040778 response_code = 500 uri = /omicron-zones
7572024-05-14T09:39:45.622ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "cfb226ee-3cb4-4e20-8e39-2b76e9040778", "content-length": "124", "date": "Tue, 14 May 2024 09:39:44 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "cfb226ee-3cb4-4e20-8e39-2b76e9040778" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 98.79746ms sled_address = [fd00:1122:3344:101::1]:12345
7582024-05-14T09:39:45.722ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
7592024-05-14T09:39:45.723ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
7602024-05-14T09:39:45.723ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
7612024-05-14T09:39:45.743ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
7622024-05-14T09:39:45.743ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7632024-05-14T09:39:45.763ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
7642024-05-14T09:39:45.763ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7652024-05-14T09:39:45.781ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7662024-05-14T09:39:45.781ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7672024-05-14T09:39:45.801ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7682024-05-14T09:39:45.801ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7692024-05-14T09:39:45.820ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7702024-05-14T09:39:45.820ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7712024-05-14T09:39:45.840ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7722024-05-14T09:39:45.840ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7732024-05-14T09:39:45.859ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7742024-05-14T09:39:45.859ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7752024-05-14T09:39:45.879ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
7762024-05-14T09:39:45.879ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
7772024-05-14T09:39:45.879ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
7782024-05-14T09:39:45.879ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
7792024-05-14T09:39:46.004ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:801
7802024-05-14T09:39:46.011ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
7812024-05-14T09:39:46.011ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
7822024-05-14T09:39:46.011ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
7832024-05-14T09:39:46.011ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
7842024-05-14T09:39:46.011ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
7852024-05-14T09:39:46.011ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265", TimeNotSynchronized), ("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d", TimeNotSynchronized), ("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997", TimeNotSynchronized), ("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf", TimeNotSynchronized), ("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 190943 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59710 req_id = 5e135159-0c10-41d2-bdf8-a297d1adbd85 response_code = 500 uri = /omicron-zones
7862024-05-14T09:39:46.012ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "5e135159-0c10-41d2-bdf8-a297d1adbd85", "content-length": "124", "date": "Tue, 14 May 2024 09:39:46 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "5e135159-0c10-41d2-bdf8-a297d1adbd85" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 307.235663ms sled_address = [fd00:1122:3344:101::1]:12345
7872024-05-14T09:39:46.320ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
7882024-05-14T09:39:46.321ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
7892024-05-14T09:39:46.321ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
7902024-05-14T09:39:46.342ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
7912024-05-14T09:39:46.342ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7922024-05-14T09:39:46.362ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
7932024-05-14T09:39:46.362ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7942024-05-14T09:39:46.381ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7952024-05-14T09:39:46.381ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7962024-05-14T09:39:46.401ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7972024-05-14T09:39:46.401ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7982024-05-14T09:39:46.419ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7992024-05-14T09:39:46.419ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8002024-05-14T09:39:46.438ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8012024-05-14T09:39:46.438ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8022024-05-14T09:39:46.458ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8032024-05-14T09:39:46.458ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8042024-05-14T09:39:46.477ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
8052024-05-14T09:39:46.477ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
8062024-05-14T09:39:46.477ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
8072024-05-14T09:39:46.477ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
8082024-05-14T09:39:46.511ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
8092024-05-14T09:39:46.511ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
8102024-05-14T09:39:46.511ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
8112024-05-14T09:39:46.511ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
8122024-05-14T09:39:46.511ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
8132024-05-14T09:39:46.511ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997", TimeNotSynchronized), ("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d", TimeNotSynchronized), ("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529", TimeNotSynchronized), ("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf", TimeNotSynchronized), ("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 190488 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59710 req_id = 3806c413-1a73-4cde-b766-57a9308e24d2 response_code = 500 uri = /omicron-zones
8142024-05-14T09:39:46.511ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "3806c413-1a73-4cde-b766-57a9308e24d2", "content-length": "124", "date": "Tue, 14 May 2024 09:39:46 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "3806c413-1a73-4cde-b766-57a9308e24d2" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 354.440447ms sled_address = [fd00:1122:3344:101::1]:12345
8152024-05-14T09:39:47.268ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
8162024-05-14T09:39:47.268ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8172024-05-14T09:39:47.268ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
8182024-05-14T09:39:47.272ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:801
8192024-05-14T09:39:47.289ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8202024-05-14T09:39:47.290ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8212024-05-14T09:39:47.308ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8222024-05-14T09:39:47.308ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8232024-05-14T09:39:47.326ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8242024-05-14T09:39:47.326ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8252024-05-14T09:39:47.346ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8262024-05-14T09:39:47.346ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8272024-05-14T09:39:47.364ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8282024-05-14T09:39:47.364ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8292024-05-14T09:39:47.383ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8302024-05-14T09:39:47.383ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8312024-05-14T09:39:47.401ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8322024-05-14T09:39:47.401ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8332024-05-14T09:39:47.419ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
8342024-05-14T09:39:47.419ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
8352024-05-14T09:39:47.419ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
8362024-05-14T09:39:47.419ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
8372024-05-14T09:39:47.452ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
8382024-05-14T09:39:47.452ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
8392024-05-14T09:39:47.452ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
8402024-05-14T09:39:47.452ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
8412024-05-14T09:39:47.452ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
8422024-05-14T09:39:47.452ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529", TimeNotSynchronized), ("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf", TimeNotSynchronized), ("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d", TimeNotSynchronized), ("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265", TimeNotSynchronized), ("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 183474 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59710 req_id = c8900546-87fc-42d0-877b-ca0b52554360 response_code = 500 uri = /omicron-zones
8432024-05-14T09:39:47.452ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "c8900546-87fc-42d0-877b-ca0b52554360", "content-length": "124", "date": "Tue, 14 May 2024 09:39:47 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "c8900546-87fc-42d0-877b-ca0b52554360" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 333.901955ms sled_address = [fd00:1122:3344:101::1]:12345
8442024-05-14T09:39:47.787ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
8452024-05-14T09:39:47.788ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8462024-05-14T09:39:47.788ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
8472024-05-14T09:39:47.808ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8482024-05-14T09:39:47.808ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8492024-05-14T09:39:47.826ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8502024-05-14T09:39:47.826ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8512024-05-14T09:39:47.846ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8522024-05-14T09:39:47.846ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8532024-05-14T09:39:47.864ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8542024-05-14T09:39:47.864ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8552024-05-14T09:39:47.882ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8562024-05-14T09:39:47.882ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8572024-05-14T09:39:47.902ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8582024-05-14T09:39:47.902ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8592024-05-14T09:39:47.920ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8602024-05-14T09:39:47.920ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8612024-05-14T09:39:47.938ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
8622024-05-14T09:39:47.938ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
8632024-05-14T09:39:47.938ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
8642024-05-14T09:39:47.938ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
8652024-05-14T09:39:47.970ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
8662024-05-14T09:39:47.971ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
8672024-05-14T09:39:47.971ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
8682024-05-14T09:39:47.971ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
8692024-05-14T09:39:47.971ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
8702024-05-14T09:39:47.971ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529", TimeNotSynchronized), ("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265", TimeNotSynchronized), ("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d", TimeNotSynchronized), ("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf", TimeNotSynchronized), ("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 183022 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59710 req_id = dad4e0da-311c-43e8-8c6b-548b20247a17 response_code = 500 uri = /omicron-zones
8712024-05-14T09:39:47.971ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "dad4e0da-311c-43e8-8c6b-548b20247a17", "content-length": "124", "date": "Tue, 14 May 2024 09:39:47 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "dad4e0da-311c-43e8-8c6b-548b20247a17" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 415.247873ms sled_address = [fd00:1122:3344:101::1]:12345
8722024-05-14T09:39:48.388ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
8732024-05-14T09:39:48.389ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8742024-05-14T09:39:48.389ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
8752024-05-14T09:39:48.410ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8762024-05-14T09:39:48.410ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8772024-05-14T09:39:48.429ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8782024-05-14T09:39:48.429ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8792024-05-14T09:39:48.449ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8802024-05-14T09:39:48.449ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8812024-05-14T09:39:48.468ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8822024-05-14T09:39:48.468ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8832024-05-14T09:39:48.488ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8842024-05-14T09:39:48.488ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8852024-05-14T09:39:48.506ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8862024-05-14T09:39:48.506ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8872024-05-14T09:39:48.526ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8882024-05-14T09:39:48.526ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8892024-05-14T09:39:48.545ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
8902024-05-14T09:39:48.545ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
8912024-05-14T09:39:48.545ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
8922024-05-14T09:39:48.545ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
8932024-05-14T09:39:48.579ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
8942024-05-14T09:39:48.579ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
8952024-05-14T09:39:48.579ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
8962024-05-14T09:39:48.579ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
8972024-05-14T09:39:48.579ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
8982024-05-14T09:39:48.579ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997", TimeNotSynchronized), ("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529", TimeNotSynchronized), ("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265", TimeNotSynchronized), ("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf", TimeNotSynchronized), ("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 191224 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59710 req_id = 070ffbe9-e787-42e1-a9df-2dd78ed05a13 response_code = 500 uri = /omicron-zones
8992024-05-14T09:39:48.579ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "070ffbe9-e787-42e1-a9df-2dd78ed05a13", "content-length": "124", "date": "Tue, 14 May 2024 09:39:48 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "070ffbe9-e787-42e1-a9df-2dd78ed05a13" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 362.34392ms sled_address = [fd00:1122:3344:101::1]:12345
9002024-05-14T09:39:48.943ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
9012024-05-14T09:39:48.943ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9022024-05-14T09:39:48.943ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
9032024-05-14T09:39:48.962ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9042024-05-14T09:39:48.963ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9052024-05-14T09:39:48.981ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9062024-05-14T09:39:48.981ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9072024-05-14T09:39:48.999ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9082024-05-14T09:39:48.999ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9092024-05-14T09:39:49.019ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9102024-05-14T09:39:49.019ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9112024-05-14T09:39:49.037ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9122024-05-14T09:39:49.037ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9132024-05-14T09:39:49.057ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9142024-05-14T09:39:49.057ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9152024-05-14T09:39:49.075ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9162024-05-14T09:39:49.075ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9172024-05-14T09:39:49.094ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
9182024-05-14T09:39:49.094ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
9192024-05-14T09:39:49.094ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
9202024-05-14T09:39:49.094ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
9212024-05-14T09:39:49.128ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
9222024-05-14T09:39:49.128ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
9232024-05-14T09:39:49.128ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
9242024-05-14T09:39:49.128ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
9252024-05-14T09:39:49.128ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
9262024-05-14T09:39:49.128ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d", TimeNotSynchronized), ("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265", TimeNotSynchronized), ("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529", TimeNotSynchronized), ("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf", TimeNotSynchronized), ("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 185138 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59710 req_id = 47c55e30-c9ba-47da-8edc-61414da7743b response_code = 500 uri = /omicron-zones
9272024-05-14T09:39:49.128ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "47c55e30-c9ba-47da-8edc-61414da7743b", "content-length": "124", "date": "Tue, 14 May 2024 09:39:49 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "47c55e30-c9ba-47da-8edc-61414da7743b" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 520.274602ms sled_address = [fd00:1122:3344:101::1]:12345
9282024-05-14T09:39:49.650ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
9292024-05-14T09:39:49.650ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9302024-05-14T09:39:49.650ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
9312024-05-14T09:39:49.670ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9322024-05-14T09:39:49.670ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9332024-05-14T09:39:49.689ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9342024-05-14T09:39:49.689ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9352024-05-14T09:39:49.707ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9362024-05-14T09:39:49.707ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9372024-05-14T09:39:49.726ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9382024-05-14T09:39:49.726ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9392024-05-14T09:39:49.745ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9402024-05-14T09:39:49.745ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9412024-05-14T09:39:49.763ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9422024-05-14T09:39:49.763ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9432024-05-14T09:39:49.782ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9442024-05-14T09:39:49.782ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9452024-05-14T09:39:49.800ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
9462024-05-14T09:39:49.800ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
9472024-05-14T09:39:49.801ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
9482024-05-14T09:39:49.801ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
9492024-05-14T09:39:49.832ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
9502024-05-14T09:39:49.833ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
9512024-05-14T09:39:49.833ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
9522024-05-14T09:39:49.833ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
9532024-05-14T09:39:49.833ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
9542024-05-14T09:39:49.833ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265", TimeNotSynchronized), ("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529", TimeNotSynchronized), ("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d", TimeNotSynchronized), ("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf", TimeNotSynchronized), ("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 182775 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59710 req_id = 972054ce-b943-415b-b73f-c0256e8fa3fd response_code = 500 uri = /omicron-zones
9552024-05-14T09:39:49.833ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "972054ce-b943-415b-b73f-c0256e8fa3fd", "content-length": "124", "date": "Tue, 14 May 2024 09:39:49 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "972054ce-b943-415b-b73f-c0256e8fa3fd" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 607.373731ms sled_address = [fd00:1122:3344:101::1]:12345
9562024-05-14T09:39:50.441ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
9572024-05-14T09:39:50.442ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9582024-05-14T09:39:50.442ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
9592024-05-14T09:39:50.463ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9602024-05-14T09:39:50.463ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9612024-05-14T09:39:50.481ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9622024-05-14T09:39:50.481ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9632024-05-14T09:39:50.499ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9642024-05-14T09:39:50.499ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9652024-05-14T09:39:50.519ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9662024-05-14T09:39:50.519ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9672024-05-14T09:39:50.538ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9682024-05-14T09:39:50.538ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9692024-05-14T09:39:50.557ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9702024-05-14T09:39:50.557ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9712024-05-14T09:39:50.575ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9722024-05-14T09:39:50.575ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9732024-05-14T09:39:50.594ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
9742024-05-14T09:39:50.594ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
9752024-05-14T09:39:50.594ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
9762024-05-14T09:39:50.594ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
9772024-05-14T09:39:50.629ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
9782024-05-14T09:39:50.629ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
9792024-05-14T09:39:50.629ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
9802024-05-14T09:39:50.629ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
9812024-05-14T09:39:50.629ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
9822024-05-14T09:39:50.630ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529", TimeNotSynchronized), ("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d", TimeNotSynchronized), ("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf", TimeNotSynchronized), ("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265", TimeNotSynchronized), ("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 187994 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59710 req_id = 71d47320-6b19-4d98-9432-b8ee4c4448a3 response_code = 500 uri = /omicron-zones
9832024-05-14T09:39:50.630ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "71d47320-6b19-4d98-9432-b8ee4c4448a3", "content-length": "124", "date": "Tue, 14 May 2024 09:39:50 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "71d47320-6b19-4d98-9432-b8ee4c4448a3" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 1.099324277s sled_address = [fd00:1122:3344:101::1]:12345
9842024-05-14T09:39:51.730ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
9852024-05-14T09:39:51.731ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9862024-05-14T09:39:51.731ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
9872024-05-14T09:39:51.751ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9882024-05-14T09:39:51.751ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9892024-05-14T09:39:51.769ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9902024-05-14T09:39:51.769ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9912024-05-14T09:39:51.787ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9922024-05-14T09:39:51.787ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9932024-05-14T09:39:51.805ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9942024-05-14T09:39:51.805ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9952024-05-14T09:39:51.824ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9962024-05-14T09:39:51.824ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9972024-05-14T09:39:51.842ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9982024-05-14T09:39:51.842ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9992024-05-14T09:39:51.860ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10002024-05-14T09:39:51.860ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10012024-05-14T09:39:51.878ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
10022024-05-14T09:39:51.879ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
10032024-05-14T09:39:51.879ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
10042024-05-14T09:39:51.879ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
10052024-05-14T09:39:51.912ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
10062024-05-14T09:39:51.912ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
10072024-05-14T09:39:51.912ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
10082024-05-14T09:39:51.912ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
10092024-05-14T09:39:51.912ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
10102024-05-14T09:39:51.912ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265", TimeNotSynchronized), ("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997", TimeNotSynchronized), ("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d", TimeNotSynchronized), ("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529", TimeNotSynchronized), ("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 181708 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59710 req_id = 309208b4-eecd-43a4-8f61-c5cfe1944eca response_code = 500 uri = /omicron-zones
10112024-05-14T09:39:51.912ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "309208b4-eecd-43a4-8f61-c5cfe1944eca", "content-length": "124", "date": "Tue, 14 May 2024 09:39:51 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "309208b4-eecd-43a4-8f61-c5cfe1944eca" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 1.225792066s sled_address = [fd00:1122:3344:101::1]:12345
10122024-05-14T09:39:53.140ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
10132024-05-14T09:39:53.140ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10142024-05-14T09:39:53.140ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
10152024-05-14T09:39:53.164ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10162024-05-14T09:39:53.164ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10172024-05-14T09:39:53.185ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10182024-05-14T09:39:53.185ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10192024-05-14T09:39:53.205ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10202024-05-14T09:39:53.205ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10212024-05-14T09:39:53.224ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10222024-05-14T09:39:53.224ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10232024-05-14T09:39:53.244ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10242024-05-14T09:39:53.244ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10252024-05-14T09:39:53.263ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10262024-05-14T09:39:53.263ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10272024-05-14T09:39:53.282ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10282024-05-14T09:39:53.282ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10292024-05-14T09:39:53.303ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
10302024-05-14T09:39:53.303ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
10312024-05-14T09:39:53.303ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
10322024-05-14T09:39:53.303ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
10332024-05-14T09:39:53.339ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
10342024-05-14T09:39:53.339ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
10352024-05-14T09:39:53.339ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
10362024-05-14T09:39:53.339ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
10372024-05-14T09:39:53.339ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
10382024-05-14T09:39:53.339ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529", TimeNotSynchronized), ("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997", TimeNotSynchronized), ("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d", TimeNotSynchronized), ("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265", TimeNotSynchronized), ("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 199188 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59710 req_id = 060d6c7c-8f46-4609-b4d9-1b600659a1dd response_code = 500 uri = /omicron-zones
10392024-05-14T09:39:53.339ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "060d6c7c-8f46-4609-b4d9-1b600659a1dd", "content-length": "124", "date": "Tue, 14 May 2024 09:39:53 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "060d6c7c-8f46-4609-b4d9-1b600659a1dd" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 559.779472ms sled_address = [fd00:1122:3344:101::1]:12345
10402024-05-14T09:39:53.901ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
10412024-05-14T09:39:53.901ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10422024-05-14T09:39:53.901ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
10432024-05-14T09:39:53.919ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10442024-05-14T09:39:53.919ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10452024-05-14T09:39:53.936ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10462024-05-14T09:39:53.936ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10472024-05-14T09:39:53.953ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10482024-05-14T09:39:53.953ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10492024-05-14T09:39:53.972ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10502024-05-14T09:39:53.972ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10512024-05-14T09:39:53.992ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10522024-05-14T09:39:53.992ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10532024-05-14T09:39:54.010ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10542024-05-14T09:39:54.010ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10552024-05-14T09:39:54.029ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10562024-05-14T09:39:54.029ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10572024-05-14T09:39:54.049ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
10582024-05-14T09:39:54.049ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
10592024-05-14T09:39:54.049ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
10602024-05-14T09:39:54.049ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
10612024-05-14T09:39:54.083ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
10622024-05-14T09:39:54.083ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
10632024-05-14T09:39:54.083ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
10642024-05-14T09:39:54.083ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
10652024-05-14T09:39:54.083ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
10662024-05-14T09:39:54.083ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265", TimeNotSynchronized), ("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d", TimeNotSynchronized), ("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529", TimeNotSynchronized), ("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf", TimeNotSynchronized), ("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 182269 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59710 req_id = e811aeeb-1ebd-4d41-8ffd-27778c644e80 response_code = 500 uri = /omicron-zones
10672024-05-14T09:39:54.084ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "e811aeeb-1ebd-4d41-8ffd-27778c644e80", "content-length": "124", "date": "Tue, 14 May 2024 09:39:53 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "e811aeeb-1ebd-4d41-8ffd-27778c644e80" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 1.366254962s sled_address = [fd00:1122:3344:101::1]:12345
10682024-05-14T09:39:55.451ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
10692024-05-14T09:39:55.451ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10702024-05-14T09:39:55.451ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
10712024-05-14T09:39:55.476ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10722024-05-14T09:39:55.476ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10732024-05-14T09:39:55.497ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10742024-05-14T09:39:55.497ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10752024-05-14T09:39:55.517ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10762024-05-14T09:39:55.517ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10772024-05-14T09:39:55.535ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10782024-05-14T09:39:55.535ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10792024-05-14T09:39:55.555ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10802024-05-14T09:39:55.555ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10812024-05-14T09:39:55.572ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10822024-05-14T09:39:55.572ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10832024-05-14T09:39:55.592ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10842024-05-14T09:39:55.592ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10852024-05-14T09:39:55.609ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
10862024-05-14T09:39:55.609ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
10872024-05-14T09:39:55.609ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
10882024-05-14T09:39:55.609ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
10892024-05-14T09:39:55.643ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
10902024-05-14T09:39:55.643ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
10912024-05-14T09:39:55.643ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
10922024-05-14T09:39:55.643ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
10932024-05-14T09:39:55.643ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
10942024-05-14T09:39:55.643ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf", TimeNotSynchronized), ("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529", TimeNotSynchronized), ("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d", TimeNotSynchronized), ("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265", TimeNotSynchronized), ("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 191667 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59710 req_id = 098a6d24-c3b9-4eb2-b13f-07fa58e2dd8a response_code = 500 uri = /omicron-zones
10952024-05-14T09:39:55.643ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "098a6d24-c3b9-4eb2-b13f-07fa58e2dd8a", "content-length": "124", "date": "Tue, 14 May 2024 09:39:55 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "098a6d24-c3b9-4eb2-b13f-07fa58e2dd8a" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 1.533759495s sled_address = [fd00:1122:3344:101::1]:12345
10962024-05-14T09:39:57.178ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
10972024-05-14T09:39:57.179ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10982024-05-14T09:39:57.179ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
10992024-05-14T09:39:57.199ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11002024-05-14T09:39:57.199ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11012024-05-14T09:39:57.217ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11022024-05-14T09:39:57.217ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11032024-05-14T09:39:57.236ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11042024-05-14T09:39:57.236ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11052024-05-14T09:39:57.255ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11062024-05-14T09:39:57.255ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11072024-05-14T09:39:57.274ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11082024-05-14T09:39:57.274ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11092024-05-14T09:39:57.293ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11102024-05-14T09:39:57.293ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11112024-05-14T09:39:57.310ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11122024-05-14T09:39:57.310ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11132024-05-14T09:39:57.328ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
11142024-05-14T09:39:57.328ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
11152024-05-14T09:39:57.329ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
11162024-05-14T09:39:57.329ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
11172024-05-14T09:39:57.362ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
11182024-05-14T09:39:57.362ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
11192024-05-14T09:39:57.362ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
11202024-05-14T09:39:57.362ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
11212024-05-14T09:39:57.362ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
11222024-05-14T09:39:57.362ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997", TimeNotSynchronized), ("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265", TimeNotSynchronized), ("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf", TimeNotSynchronized), ("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d", TimeNotSynchronized), ("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 184252 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59710 req_id = a9cd4a5c-782a-47d8-bb23-7bc8faeda3d3 response_code = 500 uri = /omicron-zones
11232024-05-14T09:39:57.362ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "a9cd4a5c-782a-47d8-bb23-7bc8faeda3d3", "content-length": "124", "date": "Tue, 14 May 2024 09:39:57 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "a9cd4a5c-782a-47d8-bb23-7bc8faeda3d3" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 2.741833845s sled_address = [fd00:1122:3344:101::1]:12345
11242024-05-14T09:39:58.361ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 598.802377ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:426 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
11252024-05-14T09:40:00.105ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
11262024-05-14T09:40:00.106ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11272024-05-14T09:40:00.106ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
11282024-05-14T09:40:00.127ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11292024-05-14T09:40:00.127ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11302024-05-14T09:40:00.145ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11312024-05-14T09:40:00.145ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11322024-05-14T09:40:00.165ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:37350
11332024-05-14T09:40:00.165ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 81 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:37350 req_id = f4252f63-f348-40f5-a94f-a7204b682da1 response_code = 200 uri = /baseboard
11342024-05-14T09:40:00.167ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11352024-05-14T09:40:00.167ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11362024-05-14T09:40:00.186ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11372024-05-14T09:40:00.186ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11382024-05-14T09:40:00.206ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11392024-05-14T09:40:00.206ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11402024-05-14T09:40:00.227ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11412024-05-14T09:40:00.227ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11422024-05-14T09:40:00.248ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11432024-05-14T09:40:00.248ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11442024-05-14T09:40:00.268ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
11452024-05-14T09:40:00.268ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
11462024-05-14T09:40:00.268ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
11472024-05-14T09:40:00.269ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
11482024-05-14T09:40:00.302ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
11492024-05-14T09:40:00.302ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
11502024-05-14T09:40:00.302ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
11512024-05-14T09:40:00.302ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
11522024-05-14T09:40:00.302ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
11532024-05-14T09:40:00.302ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529", TimeNotSynchronized), ("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997", TimeNotSynchronized), ("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf", TimeNotSynchronized), ("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265", TimeNotSynchronized), ("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 196633 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59710 req_id = 1fb2a78b-29a3-4ab8-8db7-9688bbfb0c82 response_code = 500 uri = /omicron-zones
11542024-05-14T09:40:00.302ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "1fb2a78b-29a3-4ab8-8db7-9688bbfb0c82", "content-length": "124", "date": "Tue, 14 May 2024 09:40:00 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "1fb2a78b-29a3-4ab8-8db7-9688bbfb0c82" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 1.198845655s sled_address = [fd00:1122:3344:101::1]:12345
11552024-05-14T09:40:01.502ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
11562024-05-14T09:40:01.502ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11572024-05-14T09:40:01.502ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
11582024-05-14T09:40:01.523ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11592024-05-14T09:40:01.523ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11602024-05-14T09:40:01.541ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11612024-05-14T09:40:01.541ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11622024-05-14T09:40:01.559ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11632024-05-14T09:40:01.559ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11642024-05-14T09:40:01.578ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11652024-05-14T09:40:01.578ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11662024-05-14T09:40:01.595ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11672024-05-14T09:40:01.595ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11682024-05-14T09:40:01.613ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11692024-05-14T09:40:01.613ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11702024-05-14T09:40:01.635ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11712024-05-14T09:40:01.635ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11722024-05-14T09:40:01.653ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
11732024-05-14T09:40:01.653ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
11742024-05-14T09:40:01.653ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
11752024-05-14T09:40:01.653ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
11762024-05-14T09:40:01.687ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
11772024-05-14T09:40:01.687ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
11782024-05-14T09:40:01.687ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
11792024-05-14T09:40:01.687ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
11802024-05-14T09:40:01.687ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
11812024-05-14T09:40:01.687ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf", TimeNotSynchronized), ("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997", TimeNotSynchronized), ("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265", TimeNotSynchronized), ("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529", TimeNotSynchronized), ("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 184879 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59710 req_id = 219efba1-916a-491c-a88b-723632b53a56 response_code = 500 uri = /omicron-zones
11822024-05-14T09:40:01.687ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "219efba1-916a-491c-a88b-723632b53a56", "content-length": "124", "date": "Tue, 14 May 2024 09:40:01 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "219efba1-916a-491c-a88b-723632b53a56" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 3.194958201s sled_address = [fd00:1122:3344:101::1]:12345
11832024-05-14T09:40:04.883ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
11842024-05-14T09:40:04.883ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11852024-05-14T09:40:04.883ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
11862024-05-14T09:40:04.906ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11872024-05-14T09:40:04.906ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11882024-05-14T09:40:04.925ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11892024-05-14T09:40:04.925ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11902024-05-14T09:40:04.944ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11912024-05-14T09:40:04.944ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11922024-05-14T09:40:04.963ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11932024-05-14T09:40:04.963ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11942024-05-14T09:40:04.983ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11952024-05-14T09:40:04.983ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11962024-05-14T09:40:05.002ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11972024-05-14T09:40:05.002ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11982024-05-14T09:40:05.021ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11992024-05-14T09:40:05.021ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12002024-05-14T09:40:05.041ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
12012024-05-14T09:40:05.041ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
12022024-05-14T09:40:05.041ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
12032024-05-14T09:40:05.041ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
12042024-05-14T09:40:05.074ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
12052024-05-14T09:40:05.074ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
12062024-05-14T09:40:05.074ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
12072024-05-14T09:40:05.074ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
12082024-05-14T09:40:05.074ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
12092024-05-14T09:40:05.074ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf", TimeNotSynchronized), ("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529", TimeNotSynchronized), ("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265", TimeNotSynchronized), ("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997", TimeNotSynchronized), ("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 191137 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59710 req_id = e3c293a2-4aa0-48a8-819f-278cf45347c9 response_code = 500 uri = /omicron-zones
12102024-05-14T09:40:05.074ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "e3c293a2-4aa0-48a8-819f-278cf45347c9", "content-length": "124", "date": "Tue, 14 May 2024 09:40:05 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "e3c293a2-4aa0-48a8-819f-278cf45347c9" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 1.911824662s sled_address = [fd00:1122:3344:101::1]:12345
12112024-05-14T09:40:06.987ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
12122024-05-14T09:40:06.987ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12132024-05-14T09:40:06.987ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
12142024-05-14T09:40:07.009ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12152024-05-14T09:40:07.009ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12162024-05-14T09:40:07.033ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12172024-05-14T09:40:07.033ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12182024-05-14T09:40:07.053ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12192024-05-14T09:40:07.053ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12202024-05-14T09:40:07.072ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12212024-05-14T09:40:07.072ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12222024-05-14T09:40:07.094ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12232024-05-14T09:40:07.094ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12242024-05-14T09:40:07.112ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12252024-05-14T09:40:07.112ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12262024-05-14T09:40:07.132ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12272024-05-14T09:40:07.132ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12282024-05-14T09:40:07.151ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
12292024-05-14T09:40:07.151ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
12302024-05-14T09:40:07.151ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
12312024-05-14T09:40:07.151ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
12322024-05-14T09:40:07.185ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
12332024-05-14T09:40:07.185ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
12342024-05-14T09:40:07.185ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
12352024-05-14T09:40:07.185ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
12362024-05-14T09:40:07.185ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
12372024-05-14T09:40:07.185ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265", TimeNotSynchronized), ("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529", TimeNotSynchronized), ("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf", TimeNotSynchronized), ("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997", TimeNotSynchronized), ("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 198104 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59710 req_id = ea10be0f-6f20-493d-96af-e16469a9a5f9 response_code = 500 uri = /omicron-zones
12382024-05-14T09:40:07.185ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "ea10be0f-6f20-493d-96af-e16469a9a5f9", "content-length": "124", "date": "Tue, 14 May 2024 09:40:06 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "ea10be0f-6f20-493d-96af-e16469a9a5f9" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 2.890947956s sled_address = [fd00:1122:3344:101::1]:12345
12392024-05-14T09:40:10.077ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
12402024-05-14T09:40:10.078ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12412024-05-14T09:40:10.078ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
12422024-05-14T09:40:10.101ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12432024-05-14T09:40:10.101ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12442024-05-14T09:40:10.121ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12452024-05-14T09:40:10.121ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12462024-05-14T09:40:10.141ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12472024-05-14T09:40:10.141ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12482024-05-14T09:40:10.158ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12492024-05-14T09:40:10.159ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12502024-05-14T09:40:10.178ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12512024-05-14T09:40:10.178ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12522024-05-14T09:40:10.199ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12532024-05-14T09:40:10.199ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12542024-05-14T09:40:10.218ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12552024-05-14T09:40:10.218ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12562024-05-14T09:40:10.236ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
12572024-05-14T09:40:10.236ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
12582024-05-14T09:40:10.236ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
12592024-05-14T09:40:10.236ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
12602024-05-14T09:40:10.270ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
12612024-05-14T09:40:10.270ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
12622024-05-14T09:40:10.270ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
12632024-05-14T09:40:10.270ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
12642024-05-14T09:40:10.270ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
12652024-05-14T09:40:10.270ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf", TimeNotSynchronized), ("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d", TimeNotSynchronized), ("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997", TimeNotSynchronized), ("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529", TimeNotSynchronized), ("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 192596 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59710 req_id = 99917842-883b-4d3f-af74-cbbff0c196ee response_code = 500 uri = /omicron-zones
12662024-05-14T09:40:10.270ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "99917842-883b-4d3f-af74-cbbff0c196ee", "content-length": "124", "date": "Tue, 14 May 2024 09:40:10 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "99917842-883b-4d3f-af74-cbbff0c196ee" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 3.148046513s sled_address = [fd00:1122:3344:101::1]:12345
12672024-05-14T09:40:13.420ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
12682024-05-14T09:40:13.420ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12692024-05-14T09:40:13.420ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
12702024-05-14T09:40:13.443ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12712024-05-14T09:40:13.443ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12722024-05-14T09:40:13.463ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12732024-05-14T09:40:13.463ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12742024-05-14T09:40:13.484ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12752024-05-14T09:40:13.484ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12762024-05-14T09:40:13.501ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12772024-05-14T09:40:13.501ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12782024-05-14T09:40:13.522ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12792024-05-14T09:40:13.522ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12802024-05-14T09:40:13.542ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12812024-05-14T09:40:13.542ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12822024-05-14T09:40:13.561ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12832024-05-14T09:40:13.561ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12842024-05-14T09:40:13.580ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
12852024-05-14T09:40:13.580ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
12862024-05-14T09:40:13.580ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
12872024-05-14T09:40:13.580ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
12882024-05-14T09:40:13.612ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
12892024-05-14T09:40:13.612ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
12902024-05-14T09:40:13.612ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
12912024-05-14T09:40:13.612ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
12922024-05-14T09:40:13.612ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
12932024-05-14T09:40:13.612ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529", TimeNotSynchronized), ("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265", TimeNotSynchronized), ("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d", TimeNotSynchronized), ("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf", TimeNotSynchronized), ("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 192298 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59710 req_id = f6cf3d97-b6fc-4189-a7c2-f4566c12a2b4 response_code = 500 uri = /omicron-zones
12942024-05-14T09:40:13.612ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "f6cf3d97-b6fc-4189-a7c2-f4566c12a2b4", "content-length": "124", "date": "Tue, 14 May 2024 09:40:13 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "f6cf3d97-b6fc-4189-a7c2-f4566c12a2b4" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 4.21476287s sled_address = [fd00:1122:3344:101::1]:12345
12952024-05-14T09:40:13.996ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 1.13192185s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:426 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
12962024-05-14T09:40:17.828ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
12972024-05-14T09:40:17.829ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12982024-05-14T09:40:17.829ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
12992024-05-14T09:40:17.851ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13002024-05-14T09:40:17.851ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13012024-05-14T09:40:17.870ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13022024-05-14T09:40:17.870ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13032024-05-14T09:40:17.886ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13042024-05-14T09:40:17.886ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13052024-05-14T09:40:17.904ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13062024-05-14T09:40:17.904ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13072024-05-14T09:40:17.922ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13082024-05-14T09:40:17.922ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13092024-05-14T09:40:17.940ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13102024-05-14T09:40:17.940ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13112024-05-14T09:40:17.957ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13122024-05-14T09:40:17.957ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13132024-05-14T09:40:17.975ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
13142024-05-14T09:40:17.975ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
13152024-05-14T09:40:17.975ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
13162024-05-14T09:40:17.975ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
13172024-05-14T09:40:18.006ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
13182024-05-14T09:40:18.006ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
13192024-05-14T09:40:18.006ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
13202024-05-14T09:40:18.006ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
13212024-05-14T09:40:18.006ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
13222024-05-14T09:40:18.006ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265", TimeNotSynchronized), ("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529", TimeNotSynchronized), ("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf", TimeNotSynchronized), ("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d", TimeNotSynchronized), ("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 178036 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59710 req_id = 5315df72-2f84-4b64-86e9-5f03648923bf response_code = 500 uri = /omicron-zones
13232024-05-14T09:40:18.006ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "5315df72-2f84-4b64-86e9-5f03648923bf", "content-length": "124", "date": "Tue, 14 May 2024 09:40:18 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "5315df72-2f84-4b64-86e9-5f03648923bf" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 5.11701742s sled_address = [fd00:1122:3344:101::1]:12345
13242024-05-14T09:40:23.125ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
13252024-05-14T09:40:23.125ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13262024-05-14T09:40:23.125ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
13272024-05-14T09:40:23.148ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13282024-05-14T09:40:23.148ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13292024-05-14T09:40:23.167ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13302024-05-14T09:40:23.168ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13312024-05-14T09:40:23.185ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13322024-05-14T09:40:23.185ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13332024-05-14T09:40:23.205ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13342024-05-14T09:40:23.205ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13352024-05-14T09:40:23.223ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13362024-05-14T09:40:23.223ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13372024-05-14T09:40:23.241ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13382024-05-14T09:40:23.241ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13392024-05-14T09:40:23.259ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13402024-05-14T09:40:23.259ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13412024-05-14T09:40:23.279ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
13422024-05-14T09:40:23.279ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
13432024-05-14T09:40:23.279ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
13442024-05-14T09:40:23.279ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
13452024-05-14T09:40:23.311ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
13462024-05-14T09:40:23.311ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
13472024-05-14T09:40:23.311ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
13482024-05-14T09:40:23.311ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
13492024-05-14T09:40:23.311ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
13502024-05-14T09:40:23.311ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d", TimeNotSynchronized), ("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf", TimeNotSynchronized), ("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529", TimeNotSynchronized), ("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265", TimeNotSynchronized), ("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 186429 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59710 req_id = c3cca7c0-7fba-42b3-babc-9eb81adba179 response_code = 500 uri = /omicron-zones
13512024-05-14T09:40:23.311ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "c3cca7c0-7fba-42b3-babc-9eb81adba179", "content-length": "124", "date": "Tue, 14 May 2024 09:40:23 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "c3cca7c0-7fba-42b3-babc-9eb81adba179" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 10.717732489s sled_address = [fd00:1122:3344:101::1]:12345
13522024-05-14T09:40:30.164ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 1.967031486s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:426 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
13532024-05-14T09:40:30.197ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:50053
13542024-05-14T09:40:30.197ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 49 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:50053 req_id = cbcb1a63-91e2-4d3b-aaf4-9dd8af1a1b94 response_code = 200 uri = /baseboard
13552024-05-14T09:40:34.031ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
13562024-05-14T09:40:34.031ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13572024-05-14T09:40:34.031ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
13582024-05-14T09:40:34.054ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13592024-05-14T09:40:34.054ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13602024-05-14T09:40:34.073ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13612024-05-14T09:40:34.073ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13622024-05-14T09:40:34.091ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13632024-05-14T09:40:34.091ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13642024-05-14T09:40:34.111ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13652024-05-14T09:40:34.111ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13662024-05-14T09:40:34.130ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13672024-05-14T09:40:34.130ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13682024-05-14T09:40:34.149ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13692024-05-14T09:40:34.149ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13702024-05-14T09:40:34.166ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13712024-05-14T09:40:34.166ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13722024-05-14T09:40:34.187ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
13732024-05-14T09:40:34.187ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
13742024-05-14T09:40:34.187ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
13752024-05-14T09:40:34.188ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
13762024-05-14T09:40:34.222ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
13772024-05-14T09:40:34.222ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
13782024-05-14T09:40:34.222ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
13792024-05-14T09:40:34.222ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
13802024-05-14T09:40:34.222ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
13812024-05-14T09:40:34.222ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf", TimeNotSynchronized), ("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265", TimeNotSynchronized), ("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997", TimeNotSynchronized), ("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529", TimeNotSynchronized), ("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 191320 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59710 req_id = 26bb3c6a-88f6-41e3-b962-18db206e60cd response_code = 500 uri = /omicron-zones
13822024-05-14T09:40:34.222ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "26bb3c6a-88f6-41e3-b962-18db206e60cd", "content-length": "124", "date": "Tue, 14 May 2024 09:40:34 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "26bb3c6a-88f6-41e3-b962-18db206e60cd" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 9.730094077s sled_address = [fd00:1122:3344:101::1]:12345
13832024-05-14T09:40:43.953ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
13842024-05-14T09:40:43.954ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13852024-05-14T09:40:43.954ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
13862024-05-14T09:40:43.974ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13872024-05-14T09:40:43.974ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13882024-05-14T09:40:43.990ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13892024-05-14T09:40:43.990ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13902024-05-14T09:40:44.010ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13912024-05-14T09:40:44.010ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13922024-05-14T09:40:44.030ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13932024-05-14T09:40:44.030ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13942024-05-14T09:40:44.050ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13952024-05-14T09:40:44.050ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13962024-05-14T09:40:44.069ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13972024-05-14T09:40:44.069ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13982024-05-14T09:40:44.088ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13992024-05-14T09:40:44.088ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14002024-05-14T09:40:44.106ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
14012024-05-14T09:40:44.106ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
14022024-05-14T09:40:44.106ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
14032024-05-14T09:40:44.107ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
14042024-05-14T09:40:44.140ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
14052024-05-14T09:40:44.140ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
14062024-05-14T09:40:44.140ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
14072024-05-14T09:40:44.140ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
14082024-05-14T09:40:44.140ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
14092024-05-14T09:40:44.140ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265", TimeNotSynchronized), ("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d", TimeNotSynchronized), ("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997", TimeNotSynchronized), ("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529", TimeNotSynchronized), ("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 186872 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59710 req_id = 0a3ee484-6cf5-4eb0-89dd-15459303c3d0 response_code = 500 uri = /omicron-zones
14102024-05-14T09:40:44.141ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "0a3ee484-6cf5-4eb0-89dd-15459303c3d0", "content-length": "124", "date": "Tue, 14 May 2024 09:40:43 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "0a3ee484-6cf5-4eb0-89dd-15459303c3d0" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 13.157976652s sled_address = [fd00:1122:3344:101::1]:12345
14112024-05-14T09:40:47.166ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 3.041721484s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:426 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
14122024-05-14T09:40:57.299ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
14132024-05-14T09:40:57.299ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14142024-05-14T09:40:57.299ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
14152024-05-14T09:40:57.324ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14162024-05-14T09:40:57.324ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14172024-05-14T09:40:57.346ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14182024-05-14T09:40:57.346ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14192024-05-14T09:40:57.367ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14202024-05-14T09:40:57.367ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14212024-05-14T09:40:57.388ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14222024-05-14T09:40:57.388ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14232024-05-14T09:40:57.407ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14242024-05-14T09:40:57.407ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14252024-05-14T09:40:57.427ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14262024-05-14T09:40:57.427ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14272024-05-14T09:40:57.445ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14282024-05-14T09:40:57.445ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14292024-05-14T09:40:57.465ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
14302024-05-14T09:40:57.465ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
14312024-05-14T09:40:57.465ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
14322024-05-14T09:40:57.465ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
14332024-05-14T09:40:57.497ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
14342024-05-14T09:40:57.497ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
14352024-05-14T09:40:57.497ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
14362024-05-14T09:40:57.497ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
14372024-05-14T09:40:57.497ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
14382024-05-14T09:40:57.497ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529", TimeNotSynchronized), ("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d", TimeNotSynchronized), ("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997", TimeNotSynchronized), ("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265", TimeNotSynchronized), ("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 198568 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59710 req_id = 276a7612-afb3-4b50-838c-bd2d85272caf response_code = 500 uri = /omicron-zones
14392024-05-14T09:40:57.498ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "276a7612-afb3-4b50-838c-bd2d85272caf", "content-length": "124", "date": "Tue, 14 May 2024 09:40:57 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "276a7612-afb3-4b50-838c-bd2d85272caf" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 10.792519149s sled_address = [fd00:1122:3344:101::1]:12345
14402024-05-14T09:41:00.229ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:37369
14412024-05-14T09:41:00.229ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 45 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:37369 req_id = e01878ef-3953-4b77-8a6f-0c1b8233f415 response_code = 200 uri = /baseboard
14422024-05-14T09:41:05.241ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 6.265348524s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:426 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
14432024-05-14T09:41:08.290ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
14442024-05-14T09:41:08.290ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14452024-05-14T09:41:08.290ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
14462024-05-14T09:41:08.311ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14472024-05-14T09:41:08.311ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14482024-05-14T09:41:08.330ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14492024-05-14T09:41:08.330ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14502024-05-14T09:41:08.347ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14512024-05-14T09:41:08.347ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14522024-05-14T09:41:08.366ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14532024-05-14T09:41:08.366ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14542024-05-14T09:41:08.384ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14552024-05-14T09:41:08.384ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14562024-05-14T09:41:08.401ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14572024-05-14T09:41:08.401ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14582024-05-14T09:41:08.419ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14592024-05-14T09:41:08.419ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14602024-05-14T09:41:08.438ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
14612024-05-14T09:41:08.438ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
14622024-05-14T09:41:08.438ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
14632024-05-14T09:41:08.438ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
14642024-05-14T09:41:08.472ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
14652024-05-14T09:41:08.472ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
14662024-05-14T09:41:08.472ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
14672024-05-14T09:41:08.472ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
14682024-05-14T09:41:08.472ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
14692024-05-14T09:41:08.472ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529", TimeNotSynchronized), ("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997", TimeNotSynchronized), ("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf", TimeNotSynchronized), ("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d", TimeNotSynchronized), ("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 182140 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59710 req_id = 8ccdeb9a-35fa-4893-b8ab-55907d91e0bc response_code = 500 uri = /omicron-zones
14702024-05-14T09:41:08.472ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "8ccdeb9a-35fa-4893-b8ab-55907d91e0bc", "content-length": "124", "date": "Tue, 14 May 2024 09:41:08 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "8ccdeb9a-35fa-4893-b8ab-55907d91e0bc" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 8.671205912s sled_address = [fd00:1122:3344:101::1]:12345
14712024-05-14T09:41:17.144ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
14722024-05-14T09:41:17.144ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14732024-05-14T09:41:17.144ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
14742024-05-14T09:41:17.164ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14752024-05-14T09:41:17.164ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14762024-05-14T09:41:17.185ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14772024-05-14T09:41:17.185ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14782024-05-14T09:41:17.204ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14792024-05-14T09:41:17.204ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14802024-05-14T09:41:17.223ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14812024-05-14T09:41:17.223ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14822024-05-14T09:41:17.243ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14832024-05-14T09:41:17.243ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14842024-05-14T09:41:17.262ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14852024-05-14T09:41:17.262ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14862024-05-14T09:41:17.280ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14872024-05-14T09:41:17.280ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14882024-05-14T09:41:17.297ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
14892024-05-14T09:41:17.297ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
14902024-05-14T09:41:17.297ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
14912024-05-14T09:41:17.297ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
14922024-05-14T09:41:17.329ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
14932024-05-14T09:41:17.329ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
14942024-05-14T09:41:17.329ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
14952024-05-14T09:41:17.329ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
14962024-05-14T09:41:17.329ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
14972024-05-14T09:41:17.329ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529", TimeNotSynchronized), ("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265", TimeNotSynchronized), ("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d", TimeNotSynchronized), ("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf", TimeNotSynchronized), ("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 185234 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59710 req_id = 8e9cf4c1-c1fe-4959-9f9c-a2b57c45fab8 response_code = 500 uri = /omicron-zones
14982024-05-14T09:41:17.329ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "8e9cf4c1-c1fe-4959-9f9c-a2b57c45fab8", "content-length": "124", "date": "Tue, 14 May 2024 09:41:17 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "8e9cf4c1-c1fe-4959-9f9c-a2b57c45fab8" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 10.902692685s sled_address = [fd00:1122:3344:101::1]:12345
14992024-05-14T09:41:26.541ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 11.623793173s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:426 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
15002024-05-14T09:41:28.232ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
15012024-05-14T09:41:28.233ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15022024-05-14T09:41:28.233ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
15032024-05-14T09:41:28.254ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15042024-05-14T09:41:28.254ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15052024-05-14T09:41:28.274ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15062024-05-14T09:41:28.274ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15072024-05-14T09:41:28.293ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15082024-05-14T09:41:28.293ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15092024-05-14T09:41:28.311ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15102024-05-14T09:41:28.311ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15112024-05-14T09:41:28.330ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15122024-05-14T09:41:28.330ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15132024-05-14T09:41:28.348ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15142024-05-14T09:41:28.348ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15152024-05-14T09:41:28.366ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15162024-05-14T09:41:28.366ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15172024-05-14T09:41:28.384ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
15182024-05-14T09:41:28.384ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
15192024-05-14T09:41:28.384ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
15202024-05-14T09:41:28.384ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
15212024-05-14T09:41:28.419ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
15222024-05-14T09:41:28.419ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
15232024-05-14T09:41:28.419ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
15242024-05-14T09:41:28.419ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
15252024-05-14T09:41:28.419ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
15262024-05-14T09:41:28.419ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265", TimeNotSynchronized), ("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf", TimeNotSynchronized), ("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529", TimeNotSynchronized), ("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997", TimeNotSynchronized), ("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 187045 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59710 req_id = f9dcb85e-4a4b-43ae-bedb-db33d9518b62 response_code = 500 uri = /omicron-zones
15272024-05-14T09:41:28.419ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "f9dcb85e-4a4b-43ae-bedb-db33d9518b62", "content-length": "124", "date": "Tue, 14 May 2024 09:41:28 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "f9dcb85e-4a4b-43ae-bedb-db33d9518b62" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 17.941749623s sled_address = [fd00:1122:3344:101::1]:12345
15282024-05-14T09:41:30.259ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:50680
15292024-05-14T09:41:30.259ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 62 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:50680 req_id = 143d927a-8d4b-4625-9ac7-bbc1834ba489 response_code = 200 uri = /baseboard
15302024-05-14T09:41:46.359ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
15312024-05-14T09:41:46.359ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15322024-05-14T09:41:46.359ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
15332024-05-14T09:41:46.383ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15342024-05-14T09:41:46.383ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15352024-05-14T09:41:46.402ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15362024-05-14T09:41:46.402ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15372024-05-14T09:41:46.422ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15382024-05-14T09:41:46.422ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15392024-05-14T09:41:46.443ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15402024-05-14T09:41:46.443ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15412024-05-14T09:41:46.463ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15422024-05-14T09:41:46.463ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15432024-05-14T09:41:46.482ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15442024-05-14T09:41:46.482ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15452024-05-14T09:41:46.501ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15462024-05-14T09:41:46.501ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15472024-05-14T09:41:46.521ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
15482024-05-14T09:41:46.522ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
15492024-05-14T09:41:46.522ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
15502024-05-14T09:41:46.522ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
15512024-05-14T09:41:46.554ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
15522024-05-14T09:41:46.554ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
15532024-05-14T09:41:46.554ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
15542024-05-14T09:41:46.554ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
15552024-05-14T09:41:46.554ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
15562024-05-14T09:41:46.554ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d", TimeNotSynchronized), ("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf", TimeNotSynchronized), ("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997", TimeNotSynchronized), ("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265", TimeNotSynchronized), ("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 195063 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59710 req_id = d9eb2727-5112-45e1-a028-bd7786df951b response_code = 500 uri = /omicron-zones
15572024-05-14T09:41:46.554ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "d9eb2727-5112-45e1-a028-bd7786df951b", "content-length": "124", "date": "Tue, 14 May 2024 09:41:46 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "d9eb2727-5112-45e1-a028-bd7786df951b" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 29.184726445s sled_address = [fd00:1122:3344:101::1]:12345
15582024-05-14T09:41:53.197ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 23.342354587s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:426 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
15592024-05-14T09:42:00.287ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:39390
15602024-05-14T09:42:00.288ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 43 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:39390 req_id = 03f79863-29a2-4b58-a39d-043935c4bd2b response_code = 200 uri = /baseboard
15612024-05-14T09:42:15.736ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
15622024-05-14T09:42:15.736ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15632024-05-14T09:42:15.736ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
15642024-05-14T09:42:15.759ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15652024-05-14T09:42:15.760ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15662024-05-14T09:42:15.781ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15672024-05-14T09:42:15.781ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15682024-05-14T09:42:15.800ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15692024-05-14T09:42:15.800ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15702024-05-14T09:42:15.821ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15712024-05-14T09:42:15.821ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15722024-05-14T09:42:15.839ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15732024-05-14T09:42:15.839ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15742024-05-14T09:42:15.859ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15752024-05-14T09:42:15.859ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15762024-05-14T09:42:15.879ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15772024-05-14T09:42:15.879ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15782024-05-14T09:42:15.899ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
15792024-05-14T09:42:15.899ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
15802024-05-14T09:42:15.899ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
15812024-05-14T09:42:15.899ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
15822024-05-14T09:42:15.935ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
15832024-05-14T09:42:15.935ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
15842024-05-14T09:42:15.935ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
15852024-05-14T09:42:15.935ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
15862024-05-14T09:42:15.935ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
15872024-05-14T09:42:15.935ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf", TimeNotSynchronized), ("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529", TimeNotSynchronized), ("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997", TimeNotSynchronized), ("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d", TimeNotSynchronized), ("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 198746 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59710 req_id = ec6732ec-7dfd-4721-86bc-38d79e30fbc2 response_code = 500 uri = /omicron-zones
15882024-05-14T09:42:15.935ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "ec6732ec-7dfd-4721-86bc-38d79e30fbc2", "content-length": "124", "date": "Tue, 14 May 2024 09:42:15 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "ec6732ec-7dfd-4721-86bc-38d79e30fbc2" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 42.518667239s sled_address = [fd00:1122:3344:101::1]:12345
15892024-05-14T09:42:30.317ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:34470
15902024-05-14T09:42:30.318ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 73 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:34470 req_id = 38cfc161-b4f9-45f2-ae62-090c43c8fef4 response_code = 200 uri = /baseboard
15912024-05-14T09:42:31.572ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 79.465305238s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:426 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
15922024-05-14T09:42:58.449ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
15932024-05-14T09:42:58.449ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15942024-05-14T09:42:58.449ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
15952024-05-14T09:42:58.470ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15962024-05-14T09:42:58.470ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15972024-05-14T09:42:58.489ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15982024-05-14T09:42:58.489ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15992024-05-14T09:42:58.507ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16002024-05-14T09:42:58.507ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16012024-05-14T09:42:58.527ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16022024-05-14T09:42:58.527ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16032024-05-14T09:42:58.543ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16042024-05-14T09:42:58.543ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16052024-05-14T09:42:58.560ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16062024-05-14T09:42:58.560ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16072024-05-14T09:42:58.578ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16082024-05-14T09:42:58.578ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16092024-05-14T09:42:58.597ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
16102024-05-14T09:42:58.597ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
16112024-05-14T09:42:58.598ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
16122024-05-14T09:42:58.598ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
16132024-05-14T09:42:58.629ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
16142024-05-14T09:42:58.629ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
16152024-05-14T09:42:58.629ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
16162024-05-14T09:42:58.629ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
16172024-05-14T09:42:58.629ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
16182024-05-14T09:42:58.629ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265", TimeNotSynchronized), ("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d", TimeNotSynchronized), ("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997", TimeNotSynchronized), ("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf", TimeNotSynchronized), ("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:866 latency_us = 180294 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59710 req_id = 5ea67e25-ff50-4b14-96bf-dc405b86b51e response_code = 500 uri = /omicron-zones
16192024-05-14T09:42:58.630ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "5ea67e25-ff50-4b14-96bf-dc405b86b51e", "content-length": "124", "date": "Tue, 14 May 2024 09:42:58 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "5ea67e25-ff50-4b14-96bf-dc405b86b51e" } file = sled-agent/src/rack_setup/service.rs:468 retry_after = 41.07522185s sled_address = [fd00:1122:3344:101::1]:12345
16202024-05-14T09:43:00.347ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:50751
16212024-05-14T09:43:00.348ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 73 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:50751 req_id = 6a07dc5a-b73b-4c89-a4a2-ffb46c12b6ea response_code = 200 uri = /baseboard
16222024-05-14T09:43:30.381ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:39229
16232024-05-14T09:43:30.381ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 55 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:39229 req_id = 1797c901-1e45-441f-b84b-d5c6c9a1a7b2 response_code = 200 uri = /baseboard
16242024-05-14T09:43:39.705ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
16252024-05-14T09:43:39.706ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
16262024-05-14T09:43:39.706ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
16272024-05-14T09:43:39.729ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
16282024-05-14T09:43:39.729ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
16292024-05-14T09:43:39.749ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
16302024-05-14T09:43:39.749ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
16312024-05-14T09:43:39.769ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16322024-05-14T09:43:39.769ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16332024-05-14T09:43:39.788ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16342024-05-14T09:43:39.788ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16352024-05-14T09:43:39.806ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16362024-05-14T09:43:39.806ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16372024-05-14T09:43:39.824ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16382024-05-14T09:43:39.824ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16392024-05-14T09:43:39.844ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16402024-05-14T09:43:39.844ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16412024-05-14T09:43:39.862ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
16422024-05-14T09:43:39.862ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
16432024-05-14T09:43:39.862ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
16442024-05-14T09:43:39.862ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
16452024-05-14T09:43:39.946ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
16462024-05-14T09:43:39.946ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
16472024-05-14T09:43:39.946ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
16482024-05-14T09:43:39.946ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
16492024-05-14T09:43:39.946ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
16502024-05-14T09:43:40.043ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529 file = illumos-utils/src/zone.rs:315
16512024-05-14T09:43:40.047ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265 file = illumos-utils/src/zone.rs:315
16522024-05-14T09:43:40.051ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997 file = illumos-utils/src/zone.rs:315
16532024-05-14T09:43:40.056ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d file = illumos-utils/src/zone.rs:315
16542024-05-14T09:43:40.060ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf file = illumos-utils/src/zone.rs:315
16552024-05-14T09:43:40.065ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529 file = illumos-utils/src/zone.rs:354
16562024-05-14T09:43:40.070ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265 file = illumos-utils/src/zone.rs:354
16572024-05-14T09:43:40.074ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997 file = illumos-utils/src/zone.rs:354
16582024-05-14T09:43:40.079ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d file = illumos-utils/src/zone.rs:354
16592024-05-14T09:43:40.083ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf file = illumos-utils/src/zone.rs:354
16602024-05-14T09:43:42.725ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService6"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::5"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::5"/> <propval type="astring" name="listen_port" value="32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
16612024-05-14T09:43:42.725ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
16622024-05-14T09:43:42.734ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService7"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::7"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::7"/> <propval type="astring" name="listen_port" value="32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
16632024-05-14T09:43:42.734ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
16642024-05-14T09:43:42.739ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService5"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::3"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::3"/> <propval type="astring" name="listen_port" value="32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
16652024-05-14T09:43:42.740ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
16662024-05-14T09:43:42.867ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265") failed: Property not found. retry in 51.198061ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
16672024-05-14T09:43:42.875ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529") failed: Property not found. retry in 60.779964ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
16682024-05-14T09:43:42.881ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997") failed: Property not found. retry in 48.741053ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
16692024-05-14T09:43:42.884ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService9"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::6"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::6"/> <propval type="astring" name="listen_port" value="32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
16702024-05-14T09:43:42.884ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
16712024-05-14T09:43:42.926ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265") failed: Property not found. retry in 95.341631ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
16722024-05-14T09:43:43.045ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997") failed: Property not found. retry in 112.946537ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
16732024-05-14T09:43:43.087ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529") failed: Property not found. retry in 113.575542ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
16742024-05-14T09:43:43.283ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265") failed: Property not found. retry in 293.318699ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
16752024-05-14T09:43:43.354ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529") failed: Property not found. retry in 213.222129ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
16762024-05-14T09:43:43.425ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997") failed: Property not found. retry in 113.713364ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
16772024-05-14T09:43:43.485ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf") failed: Property not found. retry in 62.827675ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
16782024-05-14T09:43:43.496ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService8"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::4"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::4"/> <propval type="astring" name="listen_port" value="32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
16792024-05-14T09:43:43.496ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
16802024-05-14T09:43:43.557ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997") failed: Property not found. retry in 270.394394ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
16812024-05-14T09:43:43.615ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf") failed: Property not found. retry in 86.728952ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
16822024-05-14T09:43:43.651ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529") failed: Property not found. retry in 413.429308ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
16832024-05-14T09:43:43.669ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265") failed: Property not found. retry in 539.514589ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
16842024-05-14T09:43:43.718ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf") failed: Property not found. retry in 191.764577ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
16852024-05-14T09:43:43.825ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d") failed: Property not found. retry in 74.125904ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
16862024-05-14T09:43:43.846ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997") failed: Property not found. retry in 425.947729ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
16872024-05-14T09:43:43.978ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d") failed: Property not found. retry in 118.627387ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
16882024-05-14T09:43:44.374ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf") failed: Property not found. retry in 326.978848ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
16892024-05-14T09:43:44.447ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529") failed: Property not found. retry in 865.614814ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
16902024-05-14T09:43:44.587ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265") failed: Property not found. retry in 592.894659ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
16912024-05-14T09:43:44.694ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997") failed: Property not found. retry in 647.768735ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
16922024-05-14T09:43:44.762ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d") failed: Property not found. retry in 290.668197ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
16932024-05-14T09:43:44.911ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf") failed: Property not found. retry in 580.913001ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
16942024-05-14T09:43:45.084ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d") failed: Property not found. retry in 471.853883ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
16952024-05-14T09:43:45.247ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265") failed: Property not found. retry in 1.000199492s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
16962024-05-14T09:43:45.325ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529") failed: Property not found. retry in 563.797919ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
16972024-05-14T09:43:45.354ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997") failed: Property not found. retry in 971.838419ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
16982024-05-14T09:43:45.505ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf") failed: Property not found. retry in 539.984316ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
16992024-05-14T09:43:45.568ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d") failed: Property not found. retry in 807.315315ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
17002024-05-14T09:43:45.899ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529") failed: Property not found. retry in 1.031225605s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
17012024-05-14T09:43:46.055ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf") failed: Property not found. retry in 588.978639ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
17022024-05-14T09:43:46.257ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265") failed: Property not found. retry in 570.399245ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
17032024-05-14T09:43:46.335ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997") failed: Property not found. retry in 707.109269ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
17042024-05-14T09:43:46.384ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d") failed: Property not found. retry in 1.21447934s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
17052024-05-14T09:43:46.654ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf") failed: Property not found. retry in 1.474333151s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
17062024-05-14T09:43:46.836ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265") failed: Property not found. retry in 906.117695ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
17072024-05-14T09:43:46.938ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529") failed: Property not found. retry in 592.75353ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
17082024-05-14T09:43:47.051ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997") failed: Property not found. retry in 998.199928ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
17092024-05-14T09:43:47.541ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529") failed: Property not found. retry in 857.500597ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
17102024-05-14T09:43:47.608ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d") failed: Property not found. retry in 1.181997022s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
17112024-05-14T09:43:47.752ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265") failed: Property not found. retry in 591.309511ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
17122024-05-14T09:43:48.061ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997") failed: Property not found. retry in 1.169931843s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
17132024-05-14T09:43:48.140ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf") failed: Property not found. retry in 729.007643ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
17142024-05-14T09:43:48.355ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265") failed: Property not found. retry in 1.448501118s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
17152024-05-14T09:43:48.411ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529") failed: Property not found. retry in 869.438576ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
17162024-05-14T09:43:48.798ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d") failed: Property not found. retry in 1.462691171s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
17172024-05-14T09:43:48.878ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf") failed: Property not found. retry in 1.1676527s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
17182024-05-14T09:43:49.245ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997") failed: Property not found. retry in 1.295929357s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
17192024-05-14T09:43:49.294ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529") failed: Property not found. retry in 712.427192ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
17202024-05-14T09:43:49.816ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265") failed: Property not found. retry in 917.053969ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
17212024-05-14T09:43:50.015ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529") failed: Property not found. retry in 1.359798764s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
17222024-05-14T09:43:50.056ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf") failed: Property not found. retry in 729.155054ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
17232024-05-14T09:43:50.274ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d") failed: Property not found. retry in 754.442842ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
17242024-05-14T09:43:50.551ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997") failed: Property not found. retry in 1.299903021s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
17252024-05-14T09:43:50.742ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265") failed: Property not found. retry in 565.631567ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
17262024-05-14T09:43:50.796ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf") failed: Property not found. retry in 620.80952ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
17272024-05-14T09:43:51.039ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d") failed: Property not found. retry in 574.397824ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
17282024-05-14T09:43:51.319ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265") failed: Property not found. retry in 611.700492ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
17292024-05-14T09:43:51.388ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529") failed: Property not found. retry in 1.225653739s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
17302024-05-14T09:43:51.426ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf") failed: Property not found. retry in 1.497475045s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
17312024-05-14T09:43:51.863ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997") failed: Property not found. retry in 1.201129612s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
17322024-05-14T09:43:53.075ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997") failed: Property not found. retry in 1.240902912s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
17332024-05-14T09:43:54.327ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997") failed: Property not found. retry in 688.687194ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
17342024-05-14T09:43:55.223ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997") failed: Property not found. retry in 1.174529612s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
17352024-05-14T09:43:56.765ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997") failed: Property not found. retry in 826.103099ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
17362024-05-14T09:43:57.606ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997") failed: Property not found. retry in 916.551356ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
17372024-05-14T09:43:58.533ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997") failed: Property not found. retry in 531.10736ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
17382024-05-14T09:43:59.074ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997") failed: Property not found. retry in 811.847553ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
17392024-05-14T09:43:59.898ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997") failed: Property not found. retry in 1.239020149s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
17402024-05-14T09:44:00.423ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:34793
17412024-05-14T09:44:00.424ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 373 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:34793 req_id = e6af8ade-af0f-4485-bfc0-4a3cef400b55 response_code = 200 uri = /baseboard
17422024-05-14T09:44:01.147ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997") failed: Property not found. retry in 936.301647ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
17432024-05-14T09:44:02.095ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997") failed: Property not found. retry in 1.362618837s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
17442024-05-14T09:44:03.480ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_cockroachdb_dd078000-affd-4ab0-8f5f-c717d7b09529
17452024-05-14T09:44:03.480ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_cockroachdb_4e2ad775-09a8-4891-8164-63c755a82265
17462024-05-14T09:44:03.480ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_cockroachdb_d13d0066-b539-4ac5-a545-c60023ae5997
17472024-05-14T09:44:03.480ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_cockroachdb_a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d
17482024-05-14T09:44:03.480ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_cockroachdb_277c98ec-b455-4e60-8559-6222d185ceaf
17492024-05-14T09:44:03.481ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
17502024-05-14T09:44:03.482ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
17512024-05-14T09:44:03.483ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 23778259 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59710 req_id = 741caa31-9cad-4039-bdcd-32047e19fac0 response_code = 204 uri = /omicron-zones
17522024-05-14T09:44:03.528ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:61446
17532024-05-14T09:44:03.562ZINFOSledAgent (ServiceManager): Initializing CRDB Cluster - sending request to [fd00:1122:3344:101::6]:32221 file = sled-agent/src/services.rs:3437
17542024-05-14T09:44:03.930ZINFOSledAgent (ServiceManager): Formatting CRDB file = sled-agent/src/services.rs:3455
17552024-05-14T09:44:14.628ZINFOSledAgent (ServiceManager): Formatting CRDB - Completed file = sled-agent/src/services.rs:3478
17562024-05-14T09:44:14.628ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 11100720 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:61446 req_id = 62465402-dd29-40c9-87c3-f54329073ee7 response_code = 204 uri = /cockroachdb
17572024-05-14T09:44:14.640ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 126.131434258s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:426 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
17582024-05-14T09:44:14.663ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 12f3cc35-fd2a-411b-bc60-c7832113465e, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: baf23778-f6a3-4d4f-931a-1389406eba07, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2d5804f9-5e88-4859-b528-dca669659405, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd686f38-d008-435a-ab61-a03c14315355, kind: Service { id: 2d5804f9-5e88-4859-b528-dca669659405 }, name: Name("ntp-2d5804f9-5e88-4859-b528-dca669659405"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 240, 111])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: dd078000-affd-4ab0-8f5f-c717d7b09529, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4e2ad775-09a8-4891-8164-63c755a82265, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 277c98ec-b455-4e60-8559-6222d185ceaf, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: d13d0066-b539-4ac5-a545-c60023ae5997, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7d57f350-6d84-441e-aca9-427498419506, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: b5cecdd2-5810-47a4-acef-30ed4d68f165, kind: Service { id: 7d57f350-6d84-441e-aca9-427498419506 }, name: Name("external-dns-7d57f350-6d84-441e-aca9-427498419506"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 209, 89])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: bd856afa-64af-4f62-8083-7cec6f530e9b, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: c3a0d6a8-690b-4d68-9c53-ea94903a313d, kind: Service { id: bd856afa-64af-4f62-8083-7cec6f530e9b }, name: Name("external-dns-bd856afa-64af-4f62-8083-7cec6f530e9b"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 238, 230])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 1af27ceb-2d94-42b4-abb9-e7054fec8ade, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: b34dcba8-9ce7-4e76-8638-6efaa0e9a52e, kind: Service { id: 1af27ceb-2d94-42b4-abb9-e7054fec8ade }, name: Name("nexus-1af27ceb-2d94-42b4-abb9-e7054fec8ade"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 214, 74])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: d7310b7c-67e1-4338-8414-09460b7646e9, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 0e69104d-a1e3-43ae-9fa7-d0ba3efaba6d, kind: Service { id: d7310b7c-67e1-4338-8414-09460b7646e9 }, name: Name("nexus-d7310b7c-67e1-4338-8414-09460b7646e9"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 218, 154])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 10cdd4df-0856-4bcf-af87-e5757529b0c3, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: afa350c1-4cae-457d-b6fa-02b99bddeae1, kind: Service { id: 10cdd4df-0856-4bcf-af87-e5757529b0c3 }, name: Name("nexus-10cdd4df-0856-4bcf-af87-e5757529b0c3"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 145, 29])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: ec8d4748-9593-47c2-9fb9-59f95eb0cf38, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 712a9949-23a6-4a4a-97da-2917d03922ee, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: 3e918472-0cb7-49b5-a622-8308a9ca651a, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 43c965bc-d0c5-4d77-8dee-7a6da492a2da, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: beb02584-2e2a-4158-9aa0-87876e6264fa, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f204562-0cc1-49bf-9bd5-796c36691eb5, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External } } } }, OmicronZoneConfig { id: b8889290-9edc-47af-9bea-f3fc3227987f, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External } } } }, OmicronZoneConfig { id: 87eb850e-70d3-4390-9948-908831c0c747, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External } } } }, OmicronZoneConfig { id: 16207ecc-3887-4690-8f5f-8dfc1add1473, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External } } } }, OmicronZoneConfig { id: ed0566ef-ce2f-4a35-94d5-9861855ca4b5, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ae2f89f7-35c8-4c05-940c-cdee6d8f8a12 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 42b2209d-f16a-4fca-a84d-54ecc9f13877 (zpool), kind: External } } } }, OmicronZoneConfig { id: b794d8c1-39ec-4743-9237-aeb9d36e8e47, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 95ce52af-b115-45d4-acd1-3598b0bafa5d (zpool), kind: External } } } }, OmicronZoneConfig { id: 0ae414c2-10bd-42e6-9eda-6b0420abb343, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ad348a98-27df-494e-af4b-560a8a421d64 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:430 sled_address = [fd00:1122:3344:101::1]:12345
17592024-05-14T09:44:14.664ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:55458
17602024-05-14T09:44:14.665ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17612024-05-14T09:44:14.665ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 12f3cc35-fd2a-411b-bc60-c7832113465e, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: InternalDns }, responder: ... }
17622024-05-14T09:44:14.688ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17632024-05-14T09:44:14.688ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 063ecfd8-b3e0-46f6-98ae-94ccfb48c08c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: InternalDns }, responder: ... }
17642024-05-14T09:44:14.708ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17652024-05-14T09:44:14.709ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: baf23778-f6a3-4d4f-931a-1389406eba07, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: InternalDns }, responder: ... }
17662024-05-14T09:44:14.729ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17672024-05-14T09:44:14.729ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd078000-affd-4ab0-8f5f-c717d7b09529, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17682024-05-14T09:44:14.749ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17692024-05-14T09:44:14.749ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a9c1f9a0-e708-48b3-b1e9-3d9fabafd80d, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17702024-05-14T09:44:14.770ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17712024-05-14T09:44:14.770ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 4e2ad775-09a8-4891-8164-63c755a82265, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17722024-05-14T09:44:14.789ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17732024-05-14T09:44:14.789ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 277c98ec-b455-4e60-8559-6222d185ceaf, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17742024-05-14T09:44:14.810ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17752024-05-14T09:44:14.810ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d13d0066-b539-4ac5-a545-c60023ae5997, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17762024-05-14T09:44:14.829ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7d57f350-6d84-441e-aca9-427498419506, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17772024-05-14T09:44:14.829ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 7d57f350-6d84-441e-aca9-427498419506, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: ExternalDns }, responder: ... }
17782024-05-14T09:44:14.943ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd856afa-64af-4f62-8083-7cec6f530e9b, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17792024-05-14T09:44:14.943ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd856afa-64af-4f62-8083-7cec6f530e9b, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
17802024-05-14T09:44:15.049ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 712a9949-23a6-4a4a-97da-2917d03922ee, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
17812024-05-14T09:44:15.049ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 712a9949-23a6-4a4a-97da-2917d03922ee, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: Clickhouse }, responder: ... }
17822024-05-14T09:44:15.140ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: beb02584-2e2a-4158-9aa0-87876e6264fa, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17832024-05-14T09:44:15.140ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: beb02584-2e2a-4158-9aa0-87876e6264fa, dataset_name: DatasetName { pool_name: ZpoolName { id: c919f4ac-100e-4c1e-bd24-76220f4848fc (zpool), kind: External }, kind: Crucible }, responder: ... }
17842024-05-14T09:44:15.240ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f204562-0cc1-49bf-9bd5-796c36691eb5, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17852024-05-14T09:44:15.240ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f204562-0cc1-49bf-9bd5-796c36691eb5, dataset_name: DatasetName { pool_name: ZpoolName { id: 9d13b91d-5969-417a-bfea-ad89852618b9 (zpool), kind: External }, kind: Crucible }, responder: ... }
17862024-05-14T09:44:15.334ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b8889290-9edc-47af-9bea-f3fc3227987f, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17872024-05-14T09:44:15.334ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b8889290-9edc-47af-9bea-f3fc3227987f, dataset_name: DatasetName { pool_name: ZpoolName { id: 39005e7e-189e-49db-8463-a206621e7a54 (zpool), kind: External }, kind: Crucible }, responder: ... }
17882024-05-14T09:44:15.429ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 87eb850e-70d3-4390-9948-908831c0c747, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17892024-05-14T09:44:15.429ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 87eb850e-70d3-4390-9948-908831c0c747, dataset_name: DatasetName { pool_name: ZpoolName { id: a7771067-edf5-47c9-a972-69320dc9ee62 (zpool), kind: External }, kind: Crucible }, responder: ... }
17902024-05-14T09:44:15.524ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 16207ecc-3887-4690-8f5f-8dfc1add1473, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17912024-05-14T09:44:15.524ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 16207ecc-3887-4690-8f5f-8dfc1add1473, dataset_name: DatasetName { pool_name: ZpoolName { id: 096e5012-bddf-482f-8ad5-65f3b0930676 (zpool), kind: External }, kind: Crucible }, responder: ... }
17922024-05-14T09:44:15.612ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ed0566ef-ce2f-4a35-94d5-9861855ca4b5, dataset_name: DatasetName { pool_name: ZpoolName { id: ae2f89f7-35c8-4c05-940c-cdee6d8f8a12 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17932024-05-14T09:44:15.612ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ed0566ef-ce2f-4a35-94d5-9861855ca4b5, dataset_name: DatasetName { pool_name: ZpoolName { id: ae2f89f7-35c8-4c05-940c-cdee6d8f8a12 (zpool), kind: External }, kind: Crucible }, responder: ... }
17942024-05-14T09:44:15.693ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f, dataset_name: DatasetName { pool_name: ZpoolName { id: 42b2209d-f16a-4fca-a84d-54ecc9f13877 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17952024-05-14T09:44:15.694ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f, dataset_name: DatasetName { pool_name: ZpoolName { id: 42b2209d-f16a-4fca-a84d-54ecc9f13877 (zpool), kind: External }, kind: Crucible }, responder: ... }
17962024-05-14T09:44:15.762ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b794d8c1-39ec-4743-9237-aeb9d36e8e47, dataset_name: DatasetName { pool_name: ZpoolName { id: 95ce52af-b115-45d4-acd1-3598b0bafa5d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17972024-05-14T09:44:15.762ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b794d8c1-39ec-4743-9237-aeb9d36e8e47, dataset_name: DatasetName { pool_name: ZpoolName { id: 95ce52af-b115-45d4-acd1-3598b0bafa5d (zpool), kind: External }, kind: Crucible }, responder: ... }
17982024-05-14T09:44:15.848ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0ae414c2-10bd-42e6-9eda-6b0420abb343, dataset_name: DatasetName { pool_name: ZpoolName { id: ad348a98-27df-494e-af4b-560a8a421d64 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17992024-05-14T09:44:15.848ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0ae414c2-10bd-42e6-9eda-6b0420abb343, dataset_name: DatasetName { pool_name: ZpoolName { id: ad348a98-27df-494e-af4b-560a8a421d64 (zpool), kind: External }, kind: Crucible }, responder: ... }
18002024-05-14T09:44:15.943ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18012024-05-14T09:44:15.944ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
18022024-05-14T09:44:15.944ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
18032024-05-14T09:44:15.944ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18042024-05-14T09:44:16.072ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:259
18052024-05-14T09:44:16.073ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18062024-05-14T09:44:16.088ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:259
18072024-05-14T09:44:16.088ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18082024-05-14T09:44:16.088ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18092024-05-14T09:44:16.113ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:259
18102024-05-14T09:44:16.113ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18112024-05-14T09:44:16.113ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18122024-05-14T09:44:16.127ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:259
18132024-05-14T09:44:16.127ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18142024-05-14T09:44:16.127ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18152024-05-14T09:44:16.270ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:259
18162024-05-14T09:44:16.345ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18172024-05-14T09:44:16.345ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18182024-05-14T09:44:16.345ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18192024-05-14T09:44:16.345ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18202024-05-14T09:44:16.345ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18212024-05-14T09:44:16.345ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18222024-05-14T09:44:16.345ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18232024-05-14T09:44:16.345ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343 file = illumos-utils/src/zone.rs:315
18242024-05-14T09:44:16.351ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a file = illumos-utils/src/zone.rs:315
18252024-05-14T09:44:16.356ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47 file = illumos-utils/src/zone.rs:315
18262024-05-14T09:44:16.361ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da file = illumos-utils/src/zone.rs:315
18272024-05-14T09:44:16.366ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f file = illumos-utils/src/zone.rs:315
18282024-05-14T09:44:16.371ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5 file = illumos-utils/src/zone.rs:315
18292024-05-14T09:44:16.376ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38 file = illumos-utils/src/zone.rs:315
18302024-05-14T09:44:16.524ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343 file = illumos-utils/src/zone.rs:354
18312024-05-14T09:44:16.529ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a file = illumos-utils/src/zone.rs:354
18322024-05-14T09:44:16.533ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47 file = illumos-utils/src/zone.rs:354
18332024-05-14T09:44:16.538ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da file = illumos-utils/src/zone.rs:354
18342024-05-14T09:44:16.542ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f file = illumos-utils/src/zone.rs:354
18352024-05-14T09:44:16.548ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5 file = illumos-utils/src/zone.rs:354
18362024-05-14T09:44:16.552ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38 file = illumos-utils/src/zone.rs:354
18372024-05-14T09:44:16.558ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee file = illumos-utils/src/zone.rs:315
18382024-05-14T09:44:16.563ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a file = illumos-utils/src/zone.rs:315
18392024-05-14T09:44:16.568ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5 file = illumos-utils/src/zone.rs:315
18402024-05-14T09:44:16.574ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747 file = illumos-utils/src/zone.rs:315
18412024-05-14T09:44:16.580ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa file = illumos-utils/src/zone.rs:315
18422024-05-14T09:44:16.586ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473 file = illumos-utils/src/zone.rs:315
18432024-05-14T09:44:16.592ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f file = illumos-utils/src/zone.rs:315
18442024-05-14T09:44:16.600ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee file = illumos-utils/src/zone.rs:354
18452024-05-14T09:44:16.606ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a file = illumos-utils/src/zone.rs:354
18462024-05-14T09:44:16.611ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5 file = illumos-utils/src/zone.rs:354
18472024-05-14T09:44:16.616ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747 file = illumos-utils/src/zone.rs:354
18482024-05-14T09:44:16.622ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa file = illumos-utils/src/zone.rs:354
18492024-05-14T09:44:16.627ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473 file = illumos-utils/src/zone.rs:354
18502024-05-14T09:44:16.632ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f file = illumos-utils/src/zone.rs:354
18512024-05-14T09:44:16.714ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:276
18522024-05-14T09:44:16.821ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:276
18532024-05-14T09:44:16.880ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:276
18542024-05-14T09:44:16.936ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:276
18552024-05-14T09:44:16.992ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:276
18562024-05-14T09:44:16.997ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:301
18572024-05-14T09:44:16.997ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:193
18582024-05-14T09:44:17.086ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:394 port = Port { inner: PortInner { name: "opte1", ip: 172.30.2.6, mac: MacAddr6([168, 64, 37, 255, 218, 154]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 }, vnic: "vopte1" } }
18592024-05-14T09:44:17.086ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1290 zone_type = nexus
18602024-05-14T09:44:17.092ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:301
18612024-05-14T09:44:17.092ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:193
18622024-05-14T09:44:17.197ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:394 port = Port { inner: PortInner { name: "opte2", ip: 172.30.1.6, mac: MacAddr6([168, 64, 37, 255, 238, 230]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 }, vnic: "vopte2" } }
18632024-05-14T09:44:17.197ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1290 zone_type = external_dns
18642024-05-14T09:44:17.200ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:301
18652024-05-14T09:44:17.200ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:193
18662024-05-14T09:44:17.317ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:394 port = Port { inner: PortInner { name: "opte3", ip: 172.30.2.7, mac: MacAddr6([168, 64, 37, 255, 145, 29]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 }, vnic: "vopte3" } }
18672024-05-14T09:44:17.318ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1290 zone_type = nexus
18682024-05-14T09:44:17.321ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:301
18692024-05-14T09:44:17.321ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:193
18702024-05-14T09:44:17.477ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:394 port = Port { inner: PortInner { name: "opte4", ip: 172.30.1.5, mac: MacAddr6([168, 64, 37, 255, 209, 89]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 }, vnic: "vopte4" } }
18712024-05-14T09:44:17.477ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1290 zone_type = external_dns
18722024-05-14T09:44:17.480ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:301
18732024-05-14T09:44:17.480ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:193
18742024-05-14T09:44:17.581ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:394 port = Port { inner: PortInner { name: "opte5", ip: 172.30.2.5, mac: MacAddr6([168, 64, 37, 255, 214, 74]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 }, vnic: "vopte5" } }
18752024-05-14T09:44:17.581ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1290 zone_type = nexus
18762024-05-14T09:44:17.585ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.2.167, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
18772024-05-14T09:44:17.585ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.2.167, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
18782024-05-14T09:44:17.586ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.2.166, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
18792024-05-14T09:44:17.586ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.2.166, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
18802024-05-14T09:44:17.588ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.2.168, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
18812024-05-14T09:44:17.588ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.2.168, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
18822024-05-14T09:44:17.591ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.2.166, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
18832024-05-14T09:44:17.594ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18842024-05-14T09:44:17.642ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.2.167, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
18852024-05-14T09:44:17.645ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18862024-05-14T09:44:17.646ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.2.165, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
18872024-05-14T09:44:17.646ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.2.165, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
18882024-05-14T09:44:17.646ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.2.168, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
18892024-05-14T09:44:17.650ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18902024-05-14T09:44:17.650ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.2.164, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
18912024-05-14T09:44:17.650ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.2.164, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
18922024-05-14T09:44:17.697ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.2.165, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
18932024-05-14T09:44:17.734ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.2.164, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
18942024-05-14T09:44:17.737ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18952024-05-14T09:44:17.737ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18962024-05-14T09:44:17.737ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_1af27ceb-2d94-42b4-abb9-e7054fec8ade file = illumos-utils/src/zone.rs:315
18972024-05-14T09:44:17.745ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_d7310b7c-67e1-4338-8414-09460b7646e9 file = illumos-utils/src/zone.rs:315
18982024-05-14T09:44:17.796ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_10cdd4df-0856-4bcf-af87-e5757529b0c3 file = illumos-utils/src/zone.rs:315
18992024-05-14T09:44:18.055ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_1af27ceb-2d94-42b4-abb9-e7054fec8ade file = illumos-utils/src/zone.rs:354
19002024-05-14T09:44:18.104ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_d7310b7c-67e1-4338-8414-09460b7646e9 file = illumos-utils/src/zone.rs:354
19012024-05-14T09:44:18.112ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b file = illumos-utils/src/zone.rs:315
19022024-05-14T09:44:18.148ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_10cdd4df-0856-4bcf-af87-e5757529b0c3 file = illumos-utils/src/zone.rs:354
19032024-05-14T09:44:18.183ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_7d57f350-6d84-441e-aca9-427498419506 file = illumos-utils/src/zone.rs:315
19042024-05-14T09:44:18.221ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b file = illumos-utils/src/zone.rs:354
19052024-05-14T09:44:18.230ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_7d57f350-6d84-441e-aca9-427498419506 file = illumos-utils/src/zone.rs:354
19062024-05-14T09:44:21.314ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService13"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::11"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::11"/> <propval type="astring" name="listen_port" value="17000"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19072024-05-14T09:44:21.315ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da
19082024-05-14T09:44:21.663ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService18"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::10"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::10"/> <propval type="astring" name="listen_port" value="17000"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19092024-05-14T09:44:21.667ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a
19102024-05-14T09:44:21.691ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService11"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::f"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::f"/> <propval type="astring" name="listen_port" value="17000"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19112024-05-14T09:44:21.692ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a
19122024-05-14T09:44:21.765ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService10"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::1a"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_ad348a98-27df-494e-af4b-560a8a421d64/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::1a"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="571ea486-e9ba-40d0-9076-006ea67325a7"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19132024-05-14T09:44:21.890ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343
19142024-05-14T09:44:22.068ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService12"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::19"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_95ce52af-b115-45d4-acd1-3598b0bafa5d/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::19"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="1fe30370-8df1-42aa-980a-99aefba5c3ea"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19152024-05-14T09:44:22.084ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService15"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::17"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_ae2f89f7-35c8-4c05-940c-cdee6d8f8a12/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::17"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="32c57d62-aa34-4b43-8e8e-9b0351f2008f"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19162024-05-14T09:44:22.449ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService23"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::14"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_39005e7e-189e-49db-8463-a206621e7a54/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::14"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="1f55d83e-e023-463c-85d8-56a259c8064d"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19172024-05-14T09:44:22.513ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47
19182024-05-14T09:44:25.412ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5
19192024-05-14T09:44:25.490ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService19"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::13"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_9d13b91d-5969-417a-bfea-ad89852618b9/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::13"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="14178711-08b8-4e8c-9a91-8b8dfcb1126b"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19202024-05-14T09:44:25.565ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService20"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::15"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_a7771067-edf5-47c9-a972-69320dc9ee62/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::15"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="08b564d9-b71c-44b0-9080-2a55a913bfc2"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19212024-05-14T09:44:25.584ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService22"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::16"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_096e5012-bddf-482f-8ad5-65f3b0930676/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::16"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="9afafd8b-33e3-4092-a92c-b6c9ec8c1808"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19222024-05-14T09:44:25.585ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f
19232024-05-14T09:44:25.713ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5
19242024-05-14T09:44:25.919ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747
19252024-05-14T09:44:26.241ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473
19262024-05-14T09:44:26.544ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a") failed: Property not found. retry in 38.254612ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a
19272024-05-14T09:44:26.797ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da") failed: Property not found. retry in 40.605455ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da
19282024-05-14T09:44:26.798ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService14"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::18"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_42b2209d-f16a-4fca-a84d-54ecc9f13877/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::18"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="6c12baa5-4f49-4ce4-8629-0925e95d22f1"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19292024-05-14T09:44:26.963ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5") failed: Property not found. retry in 33.786321ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5
19302024-05-14T09:44:26.977ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a") failed: Property not found. retry in 61.565973ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a
19312024-05-14T09:44:26.987ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5") failed: Property not found. retry in 42.596783ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5
19322024-05-14T09:44:26.998ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747") failed: Property not found. retry in 39.071537ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747
19332024-05-14T09:44:27.011ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473") failed: Property not found. retry in 57.127074ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473
19342024-05-14T09:44:27.164ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f") failed: Property not found. retry in 48.639472ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f
19352024-05-14T09:44:27.177ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343") failed: Property not found. retry in 29.883667ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343
19362024-05-14T09:44:27.190ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a") failed: Property not found. retry in 113.605384ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a
19372024-05-14T09:44:27.203ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47") failed: Property not found. retry in 25.360959ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47
19382024-05-14T09:44:27.214ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da") failed: Property not found. retry in 125.401483ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da
19392024-05-14T09:44:27.214ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f
19402024-05-14T09:44:27.227ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5") failed: Property not found. retry in 134.863513ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5
19412024-05-14T09:44:27.238ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a") failed: Property not found. retry in 144.562753ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a
19422024-05-14T09:44:27.249ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5") failed: Property not found. retry in 117.295741ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5
19432024-05-14T09:44:27.264ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747") failed: Property not found. retry in 104.508818ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747
19442024-05-14T09:44:27.266ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService21"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::12"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_c919f4ac-100e-4c1e-bd24-76220f4848fc/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::12"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="848b45c0-8d60-4add-98c4-847c579861a2"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19452024-05-14T09:44:27.387ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473") failed: Property not found. retry in 96.816117ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473
19462024-05-14T09:44:27.399ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f") failed: Property not found. retry in 135.117651ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f
19472024-05-14T09:44:27.411ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343") failed: Property not found. retry in 84.843819ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343
19482024-05-14T09:44:27.423ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a") failed: Property not found. retry in 264.358612ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a
19492024-05-14T09:44:27.433ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47") failed: Property not found. retry in 99.038765ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47
19502024-05-14T09:44:27.445ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da") failed: Property not found. retry in 158.311777ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da
19512024-05-14T09:44:27.455ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5") failed: Property not found. retry in 143.660213ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5
19522024-05-14T09:44:27.467ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a") failed: Property not found. retry in 272.552749ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a
19532024-05-14T09:44:27.476ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5") failed: Property not found. retry in 101.940072ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5
19542024-05-14T09:44:27.487ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747") failed: Property not found. retry in 278.745188ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747
19552024-05-14T09:44:27.487ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa
19562024-05-14T09:44:27.509ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473") failed: Property not found. retry in 281.531383ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473
19572024-05-14T09:44:27.521ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343") failed: Property not found. retry in 271.24056ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343
19582024-05-14T09:44:27.548ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47") failed: Property not found. retry in 297.996417ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47
19592024-05-14T09:44:27.564ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f") failed: Property not found. retry in 116.042183ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f
19602024-05-14T09:44:27.591ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5") failed: Property not found. retry in 557.281772ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5
19612024-05-14T09:44:27.610ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5") failed: Property not found. retry in 240.536756ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5
19622024-05-14T09:44:27.624ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da") failed: Property not found. retry in 583.916849ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da
19632024-05-14T09:44:27.634ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f") failed: Property not found. retry in 35.596236ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f
19642024-05-14T09:44:27.683ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f") failed: Property not found. retry in 70.501854ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f
19652024-05-14T09:44:27.697ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f") failed: Property not found. retry in 451.463663ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f
19662024-05-14T09:44:27.709ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a") failed: Property not found. retry in 519.60125ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a
19672024-05-14T09:44:27.718ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa") failed: Property not found. retry in 46.74729ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa
19682024-05-14T09:44:27.752ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a") failed: Property not found. retry in 323.701684ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a
19692024-05-14T09:44:27.765ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f") failed: Property not found. retry in 232.787359ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f
19702024-05-14T09:44:27.777ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa") failed: Property not found. retry in 112.505624ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa
19712024-05-14T09:44:27.856ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747") failed: Property not found. retry in 519.024485ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747
19722024-05-14T09:44:27.867ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343") failed: Property not found. retry in 569.117413ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343
19732024-05-14T09:44:27.876ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47") failed: Property not found. retry in 204.588112ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47
19742024-05-14T09:44:27.887ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5") failed: Property not found. retry in 635.835267ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5
19752024-05-14T09:44:27.898ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473") failed: Property not found. retry in 245.876007ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473
19762024-05-14T09:44:27.907ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa") failed: Property not found. retry in 284.972056ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa
19772024-05-14T09:44:28.014ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f") failed: Property not found. retry in 535.450011ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f
19782024-05-14T09:44:28.089ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a") failed: Property not found. retry in 991.010289ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a
19792024-05-14T09:44:28.104ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47") failed: Property not found. retry in 1.02324291s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47
19802024-05-14T09:44:28.160ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473") failed: Property not found. retry in 856.496148ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473
19812024-05-14T09:44:28.172ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5") failed: Property not found. retry in 1.056403381s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5
19822024-05-14T09:44:28.185ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f") failed: Property not found. retry in 886.537325ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f
19832024-05-14T09:44:28.204ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa") failed: Property not found. retry in 512.067312ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa
19842024-05-14T09:44:28.220ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da") failed: Property not found. retry in 1.079767034s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da
19852024-05-14T09:44:28.241ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a") failed: Property not found. retry in 613.261093ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a
19862024-05-14T09:44:28.392ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747") failed: Property not found. retry in 709.319875ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747
19872024-05-14T09:44:28.448ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343") failed: Property not found. retry in 606.153107ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343
19882024-05-14T09:44:28.536ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5") failed: Property not found. retry in 1.378414671s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5
19892024-05-14T09:44:28.568ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f") failed: Property not found. retry in 803.253927ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f
19902024-05-14T09:44:28.732ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa") failed: Property not found. retry in 499.452734ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa
19912024-05-14T09:44:28.891ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a") failed: Property not found. retry in 643.292748ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a
19922024-05-14T09:44:29.043ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473") failed: Property not found. retry in 797.364424ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473
19932024-05-14T09:44:29.079ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343") failed: Property not found. retry in 1.277081493s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343
19942024-05-14T09:44:29.095ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f") failed: Property not found. retry in 897.174188ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f
19952024-05-14T09:44:29.110ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a") failed: Property not found. retry in 1.266293569s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a
19962024-05-14T09:44:29.125ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747") failed: Property not found. retry in 1.298591889s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747
19972024-05-14T09:44:29.143ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47") failed: Property not found. retry in 1.489514096s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47
19982024-05-14T09:44:29.243ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5") failed: Property not found. retry in 549.305986ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5
19992024-05-14T09:44:29.257ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa") failed: Property not found. retry in 875.779477ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa
20002024-05-14T09:44:29.258ZINFOSledAgent (ServiceManager): Profile for oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService16"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::d"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/oximeter"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="id" value="ec8d4748-9593-47c2-9fb9-59f95eb0cf38"/> <propval type="astring" name="address" value="[fd00:1122:3344:101::d]:12223"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
20012024-05-14T09:44:29.259ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38
20022024-05-14T09:44:29.315ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da") failed: Property not found. retry in 1.31265498s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da
20032024-05-14T09:44:29.385ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f") failed: Property not found. retry in 1.145225054s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f
20042024-05-14T09:44:29.548ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a") failed: Property not found. retry in 873.426289ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a
20052024-05-14T09:44:29.619ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38") failed: Property not found. retry in 51.932949ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38
20062024-05-14T09:44:29.691ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38") failed: Property not found. retry in 79.489205ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38
20072024-05-14T09:44:29.816ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38") failed: Property not found. retry in 206.026195ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38
20082024-05-14T09:44:29.831ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5") failed: Property not found. retry in 1.458166085s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5
20092024-05-14T09:44:29.871ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473") failed: Property not found. retry in 921.550429ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473
20102024-05-14T09:44:29.940ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5") failed: Property not found. retry in 1.436605548s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5
20112024-05-14T09:44:30.005ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f") failed: Property not found. retry in 680.142793ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f
20122024-05-14T09:44:30.034ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38") failed: Property not found. retry in 562.910467ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38
20132024-05-14T09:44:30.148ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa") failed: Property not found. retry in 1.486538383s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa
20142024-05-14T09:44:30.371ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343") failed: Property not found. retry in 1.14607985s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343
20152024-05-14T09:44:30.390ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a") failed: Property not found. retry in 821.538203ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a
20162024-05-14T09:44:30.436ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a") failed: Property not found. retry in 978.816318ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a
20172024-05-14T09:44:30.449ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747") failed: Property not found. retry in 639.018366ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747
20182024-05-14T09:44:30.546ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f") failed: Property not found. retry in 1.289693279s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f
20192024-05-14T09:44:30.611ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38") failed: Property not found. retry in 636.193082ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38
20202024-05-14T09:44:30.644ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da") failed: Property not found. retry in 1.310037449s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da
20212024-05-14T09:44:30.689ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47") failed: Property not found. retry in 681.061294ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47
20222024-05-14T09:44:30.708ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f") failed: Property not found. retry in 701.819799ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f
20232024-05-14T09:44:30.806ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473") failed: Property not found. retry in 675.358706ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473
20242024-05-14T09:44:30.889ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:58031
20252024-05-14T09:44:30.912ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 10697 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:58031 req_id = 0fd4af31-0d41-4e11-b2f8-510d5461763c response_code = 200 uri = /baseboard
20262024-05-14T09:44:31.102ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747") failed: Property not found. retry in 1.404854943s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747
20272024-05-14T09:44:31.226ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a") failed: Property not found. retry in 510.867597ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a
20282024-05-14T09:44:31.259ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38") failed: Property not found. retry in 648.918577ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38
20292024-05-14T09:44:31.304ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5") failed: Property not found. retry in 600.527282ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5
20302024-05-14T09:44:31.384ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47") failed: Property not found. retry in 1.398263056s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47
20312024-05-14T09:44:31.395ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5") failed: Property not found. retry in 1.057721923s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5
20322024-05-14T09:44:31.422ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f") failed: Property not found. retry in 1.289812465s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f
20332024-05-14T09:44:31.435ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a") failed: Property not found. retry in 711.101757ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a
20342024-05-14T09:44:31.495ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473") failed: Property not found. retry in 1.244776304s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473
20352024-05-14T09:44:31.531ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343") failed: Property not found. retry in 995.121957ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343
20362024-05-14T09:44:31.649ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa") failed: Property not found. retry in 1.480873993s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa
20372024-05-14T09:44:31.750ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a") failed: Property not found. retry in 988.085614ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a
20382024-05-14T09:44:31.850ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f") failed: Property not found. retry in 1.331714989s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f
20392024-05-14T09:44:31.920ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5") failed: Property not found. retry in 667.87703ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5
20402024-05-14T09:44:31.933ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38") failed: Property not found. retry in 1.27412751s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38
20412024-05-14T09:44:31.967ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da") failed: Property not found. retry in 1.031229818s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da
20422024-05-14T09:44:32.159ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a") failed: Property not found. retry in 1.128182057s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a
20432024-05-14T09:44:32.468ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5") failed: Property not found. retry in 1.331669096s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5
20442024-05-14T09:44:32.520ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747") failed: Property not found. retry in 541.176869ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747
20452024-05-14T09:44:32.538ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343") failed: Property not found. retry in 716.170672ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343
20462024-05-14T09:44:32.599ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5") failed: Property not found. retry in 971.09432ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5
20472024-05-14T09:44:32.724ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f") failed: Property not found. retry in 978.744348ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f
20482024-05-14T09:44:32.751ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a") failed: Property not found. retry in 786.892368ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a
20492024-05-14T09:44:32.763ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473") failed: Property not found. retry in 1.318294186s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473
20502024-05-14T09:44:32.794ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47") failed: Property not found. retry in 1.447047845s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47
20512024-05-14T09:44:33.013ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da") failed: Property not found. retry in 593.559789ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da
20522024-05-14T09:44:33.077ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747") failed: Property not found. retry in 945.57244ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747
20532024-05-14T09:44:33.145ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa") failed: Property not found. retry in 1.161628548s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa
20542024-05-14T09:44:33.195ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f") failed: Property not found. retry in 1.117845157s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f
20552024-05-14T09:44:33.223ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38") failed: Property not found. retry in 591.812117ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38
20562024-05-14T09:44:33.267ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343") failed: Property not found. retry in 1.13653135s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343
20572024-05-14T09:44:33.300ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a") failed: Property not found. retry in 1.473432137s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a
20582024-05-14T09:44:33.552ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a") failed: Property not found. retry in 1.112271773s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a
20592024-05-14T09:44:33.583ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5") failed: Property not found. retry in 819.610544ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5
20602024-05-14T09:44:33.616ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da") failed: Property not found. retry in 1.076647734s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da
20612024-05-14T09:44:33.717ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f") failed: Property not found. retry in 642.161506ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f
20622024-05-14T09:44:33.812ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5") failed: Property not found. retry in 1.212528932s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5
20632024-05-14T09:44:33.827ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38") failed: Property not found. retry in 1.401511424s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38
20642024-05-14T09:44:34.038ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747") failed: Property not found. retry in 850.765183ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747
20652024-05-14T09:44:34.096ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473") failed: Property not found. retry in 1.164320013s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473
20662024-05-14T09:44:34.252ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47") failed: Property not found. retry in 524.134941ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47
20672024-05-14T09:44:34.317ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa") failed: Property not found. retry in 1.06600067s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa
20682024-05-14T09:44:34.327ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f") failed: Property not found. retry in 1.329769162s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f
20692024-05-14T09:44:34.370ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f") failed: Property not found. retry in 1.149819961s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f
20702024-05-14T09:44:34.413ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5") failed: Property not found. retry in 1.133444815s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5
20712024-05-14T09:44:34.421ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343") failed: Property not found. retry in 631.772087ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343
20722024-05-14T09:44:34.570ZINFOSledAgent (ServiceManager): Profile for oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService17"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::e"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::e"/> <propval type="astring" name="listen_port" value="8123"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
20732024-05-14T09:44:34.594ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee
20742024-05-14T09:44:34.675ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a") failed: Property not found. retry in 605.294391ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a
20752024-05-14T09:44:34.702ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da") failed: Property not found. retry in 767.823147ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da
20762024-05-14T09:44:34.783ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a") failed: Property not found. retry in 1.201635512s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a
20772024-05-14T09:44:34.790ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47") failed: Property not found. retry in 1.005712096s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47
20782024-05-14T09:44:34.865ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee") failed: Property not found. retry in 37.527257ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee
20792024-05-14T09:44:34.897ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747") failed: Property not found. retry in 1.27523292s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747
20802024-05-14T09:44:34.910ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee") failed: Property not found. retry in 134.916086ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee
20812024-05-14T09:44:35.034ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5") failed: Property not found. retry in 659.635943ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5
20822024-05-14T09:44:35.054ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee") failed: Property not found. retry in 297.457203ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee
20832024-05-14T09:44:35.061ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343") failed: Property not found. retry in 1.303696874s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343
20842024-05-14T09:44:35.238ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38") failed: Property not found. retry in 1.041167085s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38
20852024-05-14T09:44:35.270ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473") failed: Property not found. retry in 622.392226ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473
20862024-05-14T09:44:35.290ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a") failed: Property not found. retry in 837.894273ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a
20872024-05-14T09:44:35.361ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee") failed: Property not found. retry in 217.699179ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee
20882024-05-14T09:44:35.392ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa") failed: Property not found. retry in 695.422254ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa
20892024-05-14T09:44:35.478ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da") failed: Property not found. retry in 881.791774ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da
20902024-05-14T09:44:35.529ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f") failed: Property not found. retry in 1.226761464s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f
20912024-05-14T09:44:35.554ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5") failed: Property not found. retry in 901.606903ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5
20922024-05-14T09:44:35.587ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee") failed: Property not found. retry in 627.24044ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee
20932024-05-14T09:44:35.667ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f") failed: Property not found. retry in 1.394228664s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f
20942024-05-14T09:44:35.702ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5") failed: Property not found. retry in 1.112590499s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5
20952024-05-14T09:44:35.804ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47") failed: Property not found. retry in 896.73146ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47
20962024-05-14T09:44:35.901ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473") failed: Property not found. retry in 1.492217316s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473
20972024-05-14T09:44:35.994ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a") failed: Property not found. retry in 1.114093065s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a
20982024-05-14T09:44:36.097ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa") failed: Property not found. retry in 726.585647ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa
20992024-05-14T09:44:36.158ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a") failed: Property not found. retry in 1.216298859s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a
21002024-05-14T09:44:36.182ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747") failed: Property not found. retry in 904.447438ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747
21012024-05-14T09:44:36.222ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee") failed: Property not found. retry in 1.080654734s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee
21022024-05-14T09:44:36.288ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38") failed: Property not found. retry in 1.351099838s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38
21032024-05-14T09:44:36.370ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da") failed: Property not found. retry in 768.160442ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da
21042024-05-14T09:44:36.378ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343") failed: Property not found. retry in 834.325341ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343
21052024-05-14T09:44:36.464ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5") failed: Property not found. retry in 1.035741914s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5
21062024-05-14T09:44:36.716ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47") failed: Property not found. retry in 1.054165675s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47
21072024-05-14T09:44:36.717ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_7d57f350-6d84-441e-aca9-427498419506: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService28"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::8"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte4"/> <propval type="astring" name="gateway" value="172.30.1.1"/> <propval type="astring" name="ip" value="172.30.1.5"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::8]:5353"/> <propval type="astring" name="dns_address" value="172.30.1.5:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
21082024-05-14T09:44:36.718ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService27"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::9"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte2"/> <propval type="astring" name="gateway" value="172.30.1.1"/> <propval type="astring" name="ip" value="172.30.1.6"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::9]:5353"/> <propval type="astring" name="dns_address" value="172.30.1.6:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
21092024-05-14T09:44:36.718ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_external_dns_7d57f350-6d84-441e-aca9-427498419506
21102024-05-14T09:44:36.724ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b
21112024-05-14T09:44:36.769ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f") failed: Property not found. retry in 731.337171ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f
21122024-05-14T09:44:36.832ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5") failed: Property not found. retry in 1.059744145s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5
21132024-05-14T09:44:36.851ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa") failed: Property not found. retry in 1.38364054s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa
21142024-05-14T09:44:36.979ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b") failed: Property not found. retry in 32.081627ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b
21152024-05-14T09:44:37.015ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_7d57f350-6d84-441e-aca9-427498419506") failed: Property not found. retry in 35.566944ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_7d57f350-6d84-441e-aca9-427498419506
21162024-05-14T09:44:37.023ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b") failed: Property not found. retry in 78.406173ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b
21172024-05-14T09:44:37.073ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_7d57f350-6d84-441e-aca9-427498419506") failed: Property not found. retry in 69.667401ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_7d57f350-6d84-441e-aca9-427498419506
21182024-05-14T09:44:37.106ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f") failed: Property not found. retry in 681.218494ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f
21192024-05-14T09:44:37.114ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b") failed: Property not found. retry in 107.866654ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b
21202024-05-14T09:44:37.155ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747") failed: Property not found. retry in 1.27645178s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747
21212024-05-14T09:44:37.164ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a") failed: Property not found. retry in 573.415059ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a
21222024-05-14T09:44:37.173ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da") failed: Property not found. retry in 686.349444ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da
21232024-05-14T09:44:37.183ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_7d57f350-6d84-441e-aca9-427498419506") failed: Property not found. retry in 159.045629ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_7d57f350-6d84-441e-aca9-427498419506
21242024-05-14T09:44:37.248ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343") failed: Property not found. retry in 1.263921853s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343
21252024-05-14T09:44:37.259ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b") failed: Property not found. retry in 343.404004ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b
21262024-05-14T09:44:37.259ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_10cdd4df-0856-4bcf-af87-e5757529b0c3: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService26"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::c"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte3"/> <propval type="astring" name="gateway" value="172.30.2.1"/> <propval type="astring" name="ip" value="172.30.2.7"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
21272024-05-14T09:44:37.259ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_1af27ceb-2d94-42b4-abb9-e7054fec8ade: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService24"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::a"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte5"/> <propval type="astring" name="gateway" value="172.30.2.1"/> <propval type="astring" name="ip" value="172.30.2.5"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
21282024-05-14T09:44:37.259ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_d7310b7c-67e1-4338-8414-09460b7646e9: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService25"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::b"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte1"/> <propval type="astring" name="gateway" value="172.30.2.1"/> <propval type="astring" name="ip" value="172.30.2.6"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
21292024-05-14T09:44:37.259ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_nexus_10cdd4df-0856-4bcf-af87-e5757529b0c3
21302024-05-14T09:44:37.264ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_nexus_1af27ceb-2d94-42b4-abb9-e7054fec8ade
21312024-05-14T09:44:37.269ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_nexus_d7310b7c-67e1-4338-8414-09460b7646e9
21322024-05-14T09:44:37.314ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee") failed: Property not found. retry in 1.280967274s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee
21332024-05-14T09:44:37.352ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_7d57f350-6d84-441e-aca9-427498419506") failed: Property not found. retry in 539.949484ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_7d57f350-6d84-441e-aca9-427498419506
21342024-05-14T09:44:37.385ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a") failed: Property not found. retry in 1.089431774s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a
21352024-05-14T09:44:37.405ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473") failed: Property not found. retry in 940.141031ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473
21362024-05-14T09:44:37.418ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_10cdd4df-0856-4bcf-af87-e5757529b0c3") failed: Property not found. retry in 38.771929ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_10cdd4df-0856-4bcf-af87-e5757529b0c3
21372024-05-14T09:44:37.426ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d7310b7c-67e1-4338-8414-09460b7646e9") failed: Property not found. retry in 70.032401ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d7310b7c-67e1-4338-8414-09460b7646e9
21382024-05-14T09:44:37.443ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1af27ceb-2d94-42b4-abb9-e7054fec8ade") failed: Property not found. retry in 70.450402ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1af27ceb-2d94-42b4-abb9-e7054fec8ade
21392024-05-14T09:44:37.468ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_10cdd4df-0856-4bcf-af87-e5757529b0c3") failed: Property not found. retry in 75.571555ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_10cdd4df-0856-4bcf-af87-e5757529b0c3
21402024-05-14T09:44:37.514ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d7310b7c-67e1-4338-8414-09460b7646e9") failed: Property not found. retry in 124.603541ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d7310b7c-67e1-4338-8414-09460b7646e9
21412024-05-14T09:44:37.538ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5") failed: Property not found. retry in 781.548523ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5
21422024-05-14T09:44:37.572ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f") failed: Property not found. retry in 1.278472933s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f
21432024-05-14T09:44:37.584ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_10cdd4df-0856-4bcf-af87-e5757529b0c3") failed: Property not found. retry in 241.35863ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_10cdd4df-0856-4bcf-af87-e5757529b0c3
21442024-05-14T09:44:37.596ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1af27ceb-2d94-42b4-abb9-e7054fec8ade") failed: Property not found. retry in 105.77079ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1af27ceb-2d94-42b4-abb9-e7054fec8ade
21452024-05-14T09:44:37.617ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b") failed: Property not found. retry in 905.447104ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b
21462024-05-14T09:44:37.651ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d7310b7c-67e1-4338-8414-09460b7646e9") failed: Property not found. retry in 272.314751ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d7310b7c-67e1-4338-8414-09460b7646e9
21472024-05-14T09:44:37.661ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38") failed: Property not found. retry in 1.481243961s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38
21482024-05-14T09:44:37.714ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1af27ceb-2d94-42b4-abb9-e7054fec8ade") failed: Property not found. retry in 266.185986ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1af27ceb-2d94-42b4-abb9-e7054fec8ade
21492024-05-14T09:44:37.747ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a") failed: Property not found. retry in 651.403206ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a
21502024-05-14T09:44:37.781ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47") failed: Property not found. retry in 971.176019ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47
21512024-05-14T09:44:37.798ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f") failed: Property not found. retry in 871.879023ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f
21522024-05-14T09:44:37.837ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_10cdd4df-0856-4bcf-af87-e5757529b0c3") failed: Property not found. retry in 412.678445ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_10cdd4df-0856-4bcf-af87-e5757529b0c3
21532024-05-14T09:44:37.871ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da") failed: Property not found. retry in 1.279238166s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da
21542024-05-14T09:44:37.902ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_7d57f350-6d84-441e-aca9-427498419506") failed: Property not found. retry in 1.094141758s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_7d57f350-6d84-441e-aca9-427498419506
21552024-05-14T09:44:37.913ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5") failed: Property not found. retry in 1.295595828s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5
21562024-05-14T09:44:37.932ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d7310b7c-67e1-4338-8414-09460b7646e9") failed: Property not found. retry in 528.796084ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d7310b7c-67e1-4338-8414-09460b7646e9
21572024-05-14T09:44:37.990ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1af27ceb-2d94-42b4-abb9-e7054fec8ade") failed: Property not found. retry in 431.086574ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1af27ceb-2d94-42b4-abb9-e7054fec8ade
21582024-05-14T09:44:38.246ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa") failed: Property not found. retry in 549.319533ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa
21592024-05-14T09:44:38.260ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_10cdd4df-0856-4bcf-af87-e5757529b0c3") failed: Property not found. retry in 661.796547ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_10cdd4df-0856-4bcf-af87-e5757529b0c3
21602024-05-14T09:44:38.330ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5") failed: Property not found. retry in 768.372583ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5
21612024-05-14T09:44:38.354ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473") failed: Property not found. retry in 723.348347ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473
21622024-05-14T09:44:38.409ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a") failed: Property not found. retry in 886.2387ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a
21632024-05-14T09:44:38.432ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1af27ceb-2d94-42b4-abb9-e7054fec8ade") failed: Property not found. retry in 937.698153ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1af27ceb-2d94-42b4-abb9-e7054fec8ade
21642024-05-14T09:44:38.445ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747") failed: Property not found. retry in 1.012298327s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747
21652024-05-14T09:44:38.472ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d7310b7c-67e1-4338-8414-09460b7646e9") failed: Property not found. retry in 1.084317495s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d7310b7c-67e1-4338-8414-09460b7646e9
21662024-05-14T09:44:38.485ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a") failed: Property not found. retry in 1.069402468s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a
21672024-05-14T09:44:38.522ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343") failed: Property not found. retry in 1.448411192s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343
21682024-05-14T09:44:38.533ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b") failed: Property not found. retry in 1.411241478s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b
21692024-05-14T09:44:38.608ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee") failed: Property not found. retry in 1.321455092s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee
21702024-05-14T09:44:38.683ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f") failed: Property not found. retry in 1.487485143s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f
21712024-05-14T09:44:38.764ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47") failed: Property not found. retry in 1.3634078s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47
21722024-05-14T09:44:38.805ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa") failed: Property not found. retry in 1.14015948s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa
21732024-05-14T09:44:38.863ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f") failed: Property not found. retry in 1.342026231s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f
21742024-05-14T09:44:38.934ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_10cdd4df-0856-4bcf-af87-e5757529b0c3") failed: Property not found. retry in 623.949009ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_10cdd4df-0856-4bcf-af87-e5757529b0c3
21752024-05-14T09:44:39.008ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_7d57f350-6d84-441e-aca9-427498419506") failed: Property not found. retry in 1.144366312s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_7d57f350-6d84-441e-aca9-427498419506
21762024-05-14T09:44:39.090ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473") failed: Property not found. retry in 558.538764ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473
21772024-05-14T09:44:39.110ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5") failed: Property not found. retry in 1.310084705s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5
21782024-05-14T09:44:39.152ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38") failed: Property not found. retry in 1.023945633s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38
21792024-05-14T09:44:39.162ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da") failed: Property not found. retry in 1.393954374s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da
21802024-05-14T09:44:39.221ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5") failed: Property not found. retry in 839.282853ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5
21812024-05-14T09:44:39.306ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a") failed: Property not found. retry in 1.304268409s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a
21822024-05-14T09:44:39.380ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1af27ceb-2d94-42b4-abb9-e7054fec8ade") failed: Property not found. retry in 536.214713ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1af27ceb-2d94-42b4-abb9-e7054fec8ade
21832024-05-14T09:44:39.470ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747") failed: Property not found. retry in 740.922566ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747
21842024-05-14T09:44:39.569ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a") failed: Property not found. retry in 680.860281ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a
21852024-05-14T09:44:39.586ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_10cdd4df-0856-4bcf-af87-e5757529b0c3") failed: Property not found. retry in 1.138163843s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_10cdd4df-0856-4bcf-af87-e5757529b0c3
21862024-05-14T09:44:39.600ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d7310b7c-67e1-4338-8414-09460b7646e9") failed: Property not found. retry in 604.202495ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d7310b7c-67e1-4338-8414-09460b7646e9
21872024-05-14T09:44:39.662ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473") failed: Property not found. retry in 1.077101774s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473
21882024-05-14T09:44:39.932ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1af27ceb-2d94-42b4-abb9-e7054fec8ade") failed: Property not found. retry in 1.100570906s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1af27ceb-2d94-42b4-abb9-e7054fec8ade
21892024-05-14T09:44:39.945ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee") failed: Property not found. retry in 966.629704ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee
21902024-05-14T09:44:39.957ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b") failed: Property not found. retry in 598.840695ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b
21912024-05-14T09:44:39.968ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa") failed: Property not found. retry in 726.507682ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa
21922024-05-14T09:44:39.983ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343") failed: Property not found. retry in 582.961863ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343
21932024-05-14T09:44:40.072ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5") failed: Property not found. retry in 585.490853ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5
21942024-05-14T09:44:40.143ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47") failed: Property not found. retry in 1.221008006s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47
21952024-05-14T09:44:40.171ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_7d57f350-6d84-441e-aca9-427498419506") failed: Property not found. retry in 1.111579664s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_7d57f350-6d84-441e-aca9-427498419506
21962024-05-14T09:44:40.188ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f") failed: Property not found. retry in 541.096723ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f
21972024-05-14T09:44:40.203ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38") failed: Property not found. retry in 1.221866379s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38
21982024-05-14T09:44:40.221ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d7310b7c-67e1-4338-8414-09460b7646e9") failed: Property not found. retry in 652.727739ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d7310b7c-67e1-4338-8414-09460b7646e9
21992024-05-14T09:44:40.236ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f") failed: Property not found. retry in 557.300274ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f
22002024-05-14T09:44:40.253ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747") failed: Property not found. retry in 906.215002ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747
22012024-05-14T09:44:40.272ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a") failed: Property not found. retry in 968.678ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a
22022024-05-14T09:44:40.438ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5") failed: Property not found. retry in 1.093334281s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5
22032024-05-14T09:44:40.573ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b") failed: Property not found. retry in 819.579027ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b
22042024-05-14T09:44:40.591ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da") failed: Property not found. retry in 1.194308545s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da
22052024-05-14T09:44:40.608ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343") failed: Property not found. retry in 759.337176ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343
22062024-05-14T09:44:40.631ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a") failed: Property not found. retry in 930.334413ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a
22072024-05-14T09:44:40.681ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5") failed: Property not found. retry in 558.171105ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5
22082024-05-14T09:44:40.719ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa") failed: Property not found. retry in 891.535675ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa
22092024-05-14T09:44:40.742ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_10cdd4df-0856-4bcf-af87-e5757529b0c3") failed: Property not found. retry in 868.518861ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_10cdd4df-0856-4bcf-af87-e5757529b0c3
22102024-05-14T09:44:40.760ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f") failed: Property not found. retry in 1.37693185s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f
22112024-05-14T09:44:40.782ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473") failed: Property not found. retry in 625.361526ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473
22122024-05-14T09:44:40.813ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f") failed: Property not found. retry in 689.171968ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f
22132024-05-14T09:44:40.888ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d7310b7c-67e1-4338-8414-09460b7646e9") failed: Property not found. retry in 895.717686ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d7310b7c-67e1-4338-8414-09460b7646e9
22142024-05-14T09:44:40.928ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee") failed: Property not found. retry in 722.607928ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee
22152024-05-14T09:44:41.046ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1af27ceb-2d94-42b4-abb9-e7054fec8ade") failed: Property not found. retry in 649.150046ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1af27ceb-2d94-42b4-abb9-e7054fec8ade
22162024-05-14T09:44:41.180ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747") failed: Property not found. retry in 1.023959988s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747
22172024-05-14T09:44:41.260ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5") failed: Property not found. retry in 621.917407ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5
22182024-05-14T09:44:41.281ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a") failed: Property not found. retry in 1.116307719s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a
22192024-05-14T09:44:41.303ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_7d57f350-6d84-441e-aca9-427498419506") failed: Property not found. retry in 806.909084ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_7d57f350-6d84-441e-aca9-427498419506
22202024-05-14T09:44:41.379ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47") failed: Property not found. retry in 754.679698ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47
22212024-05-14T09:44:41.391ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343") failed: Property not found. retry in 740.432645ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343
22222024-05-14T09:44:41.404ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b") failed: Property not found. retry in 1.166132589s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b
22232024-05-14T09:44:41.419ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473") failed: Property not found. retry in 1.137631939s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473
22242024-05-14T09:44:41.435ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38") failed: Property not found. retry in 858.880173ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38
22252024-05-14T09:44:41.515ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f") failed: Property not found. retry in 1.177521064s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f
22262024-05-14T09:44:41.545ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5") failed: Property not found. retry in 901.479734ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5
22272024-05-14T09:44:41.572ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a") failed: Property not found. retry in 1.119035984s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a
22282024-05-14T09:44:41.625ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_10cdd4df-0856-4bcf-af87-e5757529b0c3") failed: Property not found. retry in 1.276121102s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_10cdd4df-0856-4bcf-af87-e5757529b0c3
22292024-05-14T09:44:41.653ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa") failed: Property not found. retry in 1.206292915s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa
22302024-05-14T09:44:41.665ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee") failed: Property not found. retry in 1.293574828s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee
22312024-05-14T09:44:41.705ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1af27ceb-2d94-42b4-abb9-e7054fec8ade") failed: Property not found. retry in 675.098939ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1af27ceb-2d94-42b4-abb9-e7054fec8ade
22322024-05-14T09:44:41.798ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d7310b7c-67e1-4338-8414-09460b7646e9") failed: Property not found. retry in 862.757813ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d7310b7c-67e1-4338-8414-09460b7646e9
22332024-05-14T09:44:41.811ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da") failed: Property not found. retry in 1.147253641s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da
22342024-05-14T09:44:41.892ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5") failed: Property not found. retry in 1.444733463s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5
22352024-05-14T09:44:42.121ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_7d57f350-6d84-441e-aca9-427498419506") failed: Property not found. retry in 1.281111595s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_7d57f350-6d84-441e-aca9-427498419506
22362024-05-14T09:44:42.141ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343") failed: Property not found. retry in 611.937273ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343
22372024-05-14T09:44:42.151ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47") failed: Property not found. retry in 1.149462898s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47
22382024-05-14T09:44:42.161ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f") failed: Property not found. retry in 1.413854463s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f
22392024-05-14T09:44:42.215ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747") failed: Property not found. retry in 1.022553863s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747
22402024-05-14T09:44:42.306ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38") failed: Property not found. retry in 1.251227094s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38
22412024-05-14T09:44:42.390ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1af27ceb-2d94-42b4-abb9-e7054fec8ade") failed: Property not found. retry in 1.155583409s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1af27ceb-2d94-42b4-abb9-e7054fec8ade
22422024-05-14T09:44:42.408ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a") failed: Property not found. retry in 519.373461ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a
22432024-05-14T09:44:42.458ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5") failed: Property not found. retry in 667.16228ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5
22442024-05-14T09:44:42.568ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473") failed: Property not found. retry in 1.472350308s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473
22452024-05-14T09:44:42.580ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b") failed: Property not found. retry in 763.70895ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b
22462024-05-14T09:44:42.672ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d7310b7c-67e1-4338-8414-09460b7646e9") failed: Property not found. retry in 853.560741ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d7310b7c-67e1-4338-8414-09460b7646e9
22472024-05-14T09:44:42.701ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a") failed: Property not found. retry in 1.125709332s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a
22482024-05-14T09:44:42.711ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f") failed: Property not found. retry in 1.422787142s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f
22492024-05-14T09:44:42.762ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343") failed: Property not found. retry in 885.01484ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343
22502024-05-14T09:44:42.869ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa") failed: Property not found. retry in 1.097643526s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa
22512024-05-14T09:44:42.913ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_10cdd4df-0856-4bcf-af87-e5757529b0c3") failed: Property not found. retry in 1.160810768s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_10cdd4df-0856-4bcf-af87-e5757529b0c3
22522024-05-14T09:44:42.941ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a") failed: Property not found. retry in 1.209626938s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a
22532024-05-14T09:44:42.968ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da") failed: Property not found. retry in 999.472917ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da
22542024-05-14T09:44:42.975ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee") failed: Property not found. retry in 760.4016ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee
22552024-05-14T09:44:43.136ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5") failed: Property not found. retry in 1.40482618s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5
22562024-05-14T09:44:43.250ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747") failed: Property not found. retry in 1.202400702s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747
22572024-05-14T09:44:43.315ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47") failed: Property not found. retry in 539.104462ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47
22582024-05-14T09:44:43.354ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5") failed: Property not found. retry in 831.264007ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5
22592024-05-14T09:44:43.371ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b") failed: Property not found. retry in 1.365442933s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b
22602024-05-14T09:44:43.417ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_7d57f350-6d84-441e-aca9-427498419506") failed: Property not found. retry in 914.191638ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_7d57f350-6d84-441e-aca9-427498419506
22612024-05-14T09:44:43.538ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d7310b7c-67e1-4338-8414-09460b7646e9") failed: Property not found. retry in 674.200698ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d7310b7c-67e1-4338-8414-09460b7646e9
22622024-05-14T09:44:43.558ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1af27ceb-2d94-42b4-abb9-e7054fec8ade") failed: Property not found. retry in 1.312587682s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1af27ceb-2d94-42b4-abb9-e7054fec8ade
22632024-05-14T09:44:43.571ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38") failed: Property not found. retry in 877.168866ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38
22642024-05-14T09:44:43.587ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f") failed: Property not found. retry in 613.763258ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f
22652024-05-14T09:44:43.660ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343") failed: Property not found. retry in 1.009741096s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343
22662024-05-14T09:44:43.748ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee") failed: Property not found. retry in 675.71386ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee
22672024-05-14T09:44:43.840ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a") failed: Property not found. retry in 1.253671861s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a
22682024-05-14T09:44:43.871ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47") failed: Property not found. retry in 1.217903012s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47
22692024-05-14T09:44:43.982ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa") failed: Property not found. retry in 996.505877ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa
22702024-05-14T09:44:43.993ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da") failed: Property not found. retry in 726.536042ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da
22712024-05-14T09:44:44.056ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473") failed: Property not found. retry in 1.255891638s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473
22722024-05-14T09:44:44.088ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_10cdd4df-0856-4bcf-af87-e5757529b0c3") failed: Property not found. retry in 1.288330952s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_10cdd4df-0856-4bcf-af87-e5757529b0c3
22732024-05-14T09:44:44.148ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f") failed: Property not found. retry in 1.115121487s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f
22742024-05-14T09:44:44.162ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a") failed: Property not found. retry in 757.127427ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a
22752024-05-14T09:44:44.197ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5") failed: Property not found. retry in 765.792733ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5
22762024-05-14T09:44:44.215ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f") failed: Property not found. retry in 1.486806704s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f
22772024-05-14T09:44:44.228ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d7310b7c-67e1-4338-8414-09460b7646e9") failed: Property not found. retry in 693.999683ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d7310b7c-67e1-4338-8414-09460b7646e9
22782024-05-14T09:44:44.343ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_7d57f350-6d84-441e-aca9-427498419506") failed: Property not found. retry in 566.618676ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_7d57f350-6d84-441e-aca9-427498419506
22792024-05-14T09:44:44.440ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee") failed: Property not found. retry in 799.101475ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee
22802024-05-14T09:44:44.467ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38") failed: Property not found. retry in 734.052964ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38
22812024-05-14T09:44:44.481ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747") failed: Property not found. retry in 934.071694ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747
22822024-05-14T09:44:44.558ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5") failed: Property not found. retry in 813.14007ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5
22832024-05-14T09:44:44.684ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343") failed: Property not found. retry in 1.444220142s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343
22842024-05-14T09:44:44.731ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da") failed: Property not found. retry in 1.33287735s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da
22852024-05-14T09:44:44.752ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b") failed: Property not found. retry in 703.54994ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b
22862024-05-14T09:44:44.884ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1af27ceb-2d94-42b4-abb9-e7054fec8ade") failed: Property not found. retry in 1.100710899s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1af27ceb-2d94-42b4-abb9-e7054fec8ade
22872024-05-14T09:44:44.923ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_7d57f350-6d84-441e-aca9-427498419506") failed: Property not found. retry in 1.417605771s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_7d57f350-6d84-441e-aca9-427498419506
22882024-05-14T09:44:44.935ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a") failed: Property not found. retry in 559.981683ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a
22892024-05-14T09:44:44.945ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d7310b7c-67e1-4338-8414-09460b7646e9") failed: Property not found. retry in 1.463153257s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d7310b7c-67e1-4338-8414-09460b7646e9
22902024-05-14T09:44:44.976ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5") failed: Property not found. retry in 568.63569ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5
22912024-05-14T09:44:44.991ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa") failed: Property not found. retry in 635.231464ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa
22922024-05-14T09:44:45.103ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47") failed: Property not found. retry in 666.262402ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47
22932024-05-14T09:44:45.114ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a") failed: Property not found. retry in 1.302355292s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a
22942024-05-14T09:44:45.214ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38") failed: Property not found. retry in 1.474315948s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38
22952024-05-14T09:44:45.256ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee") failed: Property not found. retry in 1.288089358s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee
22962024-05-14T09:44:45.276ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f") failed: Property not found. retry in 1.045851905s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f
22972024-05-14T09:44:45.324ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473") failed: Property not found. retry in 885.136337ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473
22982024-05-14T09:44:45.383ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5") failed: Property not found. retry in 724.508978ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5
22992024-05-14T09:44:45.393ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_10cdd4df-0856-4bcf-af87-e5757529b0c3") failed: Property not found. retry in 1.301048826s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_10cdd4df-0856-4bcf-af87-e5757529b0c3
23002024-05-14T09:44:45.426ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747") failed: Property not found. retry in 762.495969ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747
23012024-05-14T09:44:45.467ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b") failed: Property not found. retry in 1.005227834s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b
23022024-05-14T09:44:45.506ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a") failed: Property not found. retry in 650.540077ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a
23032024-05-14T09:44:45.557ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5") failed: Property not found. retry in 719.127656ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5
23042024-05-14T09:44:45.636ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa") failed: Property not found. retry in 1.010060761s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa
23052024-05-14T09:44:45.712ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f") failed: Property not found. retry in 892.361103ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f
23062024-05-14T09:44:45.781ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47") failed: Property not found. retry in 954.911678ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47
23072024-05-14T09:44:45.999ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1af27ceb-2d94-42b4-abb9-e7054fec8ade") failed: Property not found. retry in 1.451965728s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1af27ceb-2d94-42b4-abb9-e7054fec8ade
23082024-05-14T09:44:46.075ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da") failed: Property not found. retry in 953.699737ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da
23092024-05-14T09:44:46.118ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5") failed: Property not found. retry in 724.604916ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5
23102024-05-14T09:44:46.139ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343") failed: Property not found. retry in 1.394128099s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343
23112024-05-14T09:44:46.167ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a") failed: Property not found. retry in 825.891362ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a
23122024-05-14T09:44:46.199ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747") failed: Property not found. retry in 515.067538ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747
23132024-05-14T09:44:46.320ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5") failed: Property not found. retry in 572.855273ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5
23142024-05-14T09:44:46.332ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f") failed: Property not found. retry in 555.580871ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f
23152024-05-14T09:44:46.349ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_7d57f350-6d84-441e-aca9-427498419506") failed: Property not found. retry in 677.655803ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_7d57f350-6d84-441e-aca9-427498419506
23162024-05-14T09:44:46.418ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d7310b7c-67e1-4338-8414-09460b7646e9") failed: Property not found. retry in 1.342021087s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d7310b7c-67e1-4338-8414-09460b7646e9
23172024-05-14T09:44:46.426ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a") failed: Property not found. retry in 668.068528ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a
23182024-05-14T09:44:46.481ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b") failed: Property not found. retry in 623.334043ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b
23192024-05-14T09:44:46.555ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee") failed: Property not found. retry in 1.331818581s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee
23202024-05-14T09:44:46.614ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f") failed: Property not found. retry in 976.868571ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f
23212024-05-14T09:44:46.656ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa") failed: Property not found. retry in 1.154149745s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa
23222024-05-14T09:44:46.704ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38") failed: Property not found. retry in 1.02865868s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38
23232024-05-14T09:44:46.714ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_10cdd4df-0856-4bcf-af87-e5757529b0c3") failed: Property not found. retry in 1.227649324s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_10cdd4df-0856-4bcf-af87-e5757529b0c3
23242024-05-14T09:44:46.724ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747") failed: Property not found. retry in 1.364262716s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747
23252024-05-14T09:44:46.758ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47") failed: Property not found. retry in 925.788239ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47
23262024-05-14T09:44:46.852ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5") failed: Property not found. retry in 585.592961ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5
23272024-05-14T09:44:46.897ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f") failed: Property not found. retry in 1.006361424s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f
23282024-05-14T09:44:46.905ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5") failed: Property not found. retry in 1.046431442s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5
23292024-05-14T09:44:47.001ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a") failed: Property not found. retry in 1.124846818s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a
23302024-05-14T09:44:47.036ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_7d57f350-6d84-441e-aca9-427498419506") failed: Property not found. retry in 1.289847421s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_7d57f350-6d84-441e-aca9-427498419506
23312024-05-14T09:44:47.043ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da") failed: Property not found. retry in 1.293336835s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da
23322024-05-14T09:44:47.103ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a") failed: Property not found. retry in 870.63417ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a
23332024-05-14T09:44:47.114ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b") failed: Property not found. retry in 748.100038ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b
23342024-05-14T09:44:47.285ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:801
23352024-05-14T09:44:47.447ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5") failed: Property not found. retry in 720.516334ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5
23362024-05-14T09:44:47.461ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1af27ceb-2d94-42b4-abb9-e7054fec8ade") failed: Property not found. retry in 604.27446ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1af27ceb-2d94-42b4-abb9-e7054fec8ade
23372024-05-14T09:44:47.546ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343") failed: Property not found. retry in 1.267913263s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343
23382024-05-14T09:44:47.602ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f") failed: Property not found. retry in 581.827091ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f
23392024-05-14T09:44:47.693ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47") failed: Property not found. retry in 1.095252303s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47
23402024-05-14T09:44:47.742ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38") failed: Property not found. retry in 1.276083161s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38
23412024-05-14T09:44:47.769ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d7310b7c-67e1-4338-8414-09460b7646e9") failed: Property not found. retry in 1.400847913s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d7310b7c-67e1-4338-8414-09460b7646e9
23422024-05-14T09:44:47.821ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa") failed: Property not found. retry in 1.376486405s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa
23432024-05-14T09:44:47.871ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b") failed: Property not found. retry in 687.057279ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b
23442024-05-14T09:44:47.894ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee") failed: Property not found. retry in 1.428974607s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee
23452024-05-14T09:44:47.911ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f") failed: Property not found. retry in 1.43488647s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f
23462024-05-14T09:44:47.954ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_10cdd4df-0856-4bcf-af87-e5757529b0c3") failed: Property not found. retry in 1.234721739s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_10cdd4df-0856-4bcf-af87-e5757529b0c3
23472024-05-14T09:44:47.963ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5") failed: Property not found. retry in 1.077435498s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5
23482024-05-14T09:44:47.982ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a") failed: Property not found. retry in 1.447837518s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a
23492024-05-14T09:44:48.074ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1af27ceb-2d94-42b4-abb9-e7054fec8ade") failed: Property not found. retry in 1.334897057s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1af27ceb-2d94-42b4-abb9-e7054fec8ade
23502024-05-14T09:44:48.098ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747") failed: Property not found. retry in 1.196249252s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747
23512024-05-14T09:44:48.136ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a") failed: Property not found. retry in 767.50173ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a
23522024-05-14T09:44:48.176ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5") failed: Property not found. retry in 1.155015921s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5
23532024-05-14T09:44:48.192ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f") failed: Property not found. retry in 721.470966ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f
23542024-05-14T09:44:48.337ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_7d57f350-6d84-441e-aca9-427498419506") failed: Property not found. retry in 706.333354ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_7d57f350-6d84-441e-aca9-427498419506
23552024-05-14T09:44:48.347ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da") failed: Property not found. retry in 909.655822ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da
23562024-05-14T09:44:48.568ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b") failed: Property not found. retry in 767.375398ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b
23572024-05-14T09:44:48.799ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47") failed: Property not found. retry in 1.039447499s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47
23582024-05-14T09:44:48.824ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343") failed: Property not found. retry in 1.383735442s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343
23592024-05-14T09:44:49.338ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee") failed: Property not found. retry in 1.131732892s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee
23602024-05-14T09:44:50.528ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_0ae414c2-10bd-42e6-9eda-6b0420abb343
23612024-05-14T09:44:50.528ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_external_dns_bd856afa-64af-4f62-8083-7cec6f530e9b
23622024-05-14T09:44:50.528ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_pantry_3e918472-0cb7-49b5-a622-8308a9ca651a
23632024-05-14T09:44:50.528ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_b794d8c1-39ec-4743-9237-aeb9d36e8e47
23642024-05-14T09:44:50.528ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_nexus_10cdd4df-0856-4bcf-af87-e5757529b0c3
23652024-05-14T09:44:50.528ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_pantry_43c965bc-d0c5-4d77-8dee-7a6da492a2da
23662024-05-14T09:44:50.528ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_9a7bb4b4-4ca3-47e0-82e2-2935db1acd6f
23672024-05-14T09:44:50.528ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_external_dns_7d57f350-6d84-441e-aca9-427498419506
23682024-05-14T09:44:50.528ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_ed0566ef-ce2f-4a35-94d5-9861855ca4b5
23692024-05-14T09:44:50.528ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_oximeter_ec8d4748-9593-47c2-9fb9-59f95eb0cf38
23702024-05-14T09:44:50.528ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_nexus_1af27ceb-2d94-42b4-abb9-e7054fec8ade
23712024-05-14T09:44:50.528ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_nexus_d7310b7c-67e1-4338-8414-09460b7646e9
23722024-05-14T09:44:50.528ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_clickhouse_712a9949-23a6-4a4a-97da-2917d03922ee
23732024-05-14T09:44:50.528ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_pantry_b7dc045d-c7c8-45e1-af8a-fbd0f5f7890a
23742024-05-14T09:44:50.528ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_1f204562-0cc1-49bf-9bd5-796c36691eb5
23752024-05-14T09:44:50.528ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_87eb850e-70d3-4390-9948-908831c0c747
23762024-05-14T09:44:50.528ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_beb02584-2e2a-4158-9aa0-87876e6264fa
23772024-05-14T09:44:50.529ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_16207ecc-3887-4690-8f5f-8dfc1add1473
23782024-05-14T09:44:50.529ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_b8889290-9edc-47af-9bea-f3fc3227987f
23792024-05-14T09:44:50.531ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
23802024-05-14T09:44:50.544ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
23812024-05-14T09:44:50.568ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 35904176 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:55458 req_id = 4432019e-a6ee-4357-86e5-bdeb1b7cc45d response_code = 204 uri = /omicron-zones
23822024-05-14T09:44:50.586ZINFOSledAgent (RSS): Finished setting up services file = sled-agent/src/rack_setup/service.rs:1236
23832024-05-14T09:44:50.586ZINFOSledAgent (RSS): Writing ledger to /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
23842024-05-14T09:44:50.586ZINFOSledAgent (RSS): Writing ledger to /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
23852024-05-14T09:44:50.592ZINFOSledAgent (RSS): Handing off control to Nexus file = sled-agent/src/rack_setup/service.rs:680
23862024-05-14T09:44:50.599ZINFOSledAgent (RSS): Nexus address: [fd00:1122:3344:101::c]:12221 file = sled-agent/src/rack_setup/service.rs:693
23872024-05-14T09:44:50.684ZINFOSledAgent (RSS): rack_network_config: RackNetworkConfigV1 { bfd: [], bgp: [], infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV1 { addresses: [ V4( Ipv4Network { addr: 10.151.2.174, prefix: 24, }, ), ], autoneg: false, bgp_peers: [], port: "qsfp0", routes: [ RouteConfig { destination: V4( Ipv4Network { addr: 0.0.0.0, prefix: 0, }, ), nexthop: 10.151.2.100, vlan_id: None, }, ], switch: Switch0, uplink_port_fec: None, uplink_port_speed: Speed40G, }, ], rack_subnet: Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56, }, } file = sled-agent/src/rack_setup/service.rs:817
23882024-05-14T09:44:53.621ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/240cabb9-b684-4172-8606-d31cd2279cd6/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:879
23892024-05-14T09:44:53.769ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/240cabb9-b684-4172-8606-d31cd2279cd6/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:879
23902024-05-14T09:44:53.945ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/240cabb9-b684-4172-8606-d31cd2279cd6/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:879
23912024-05-14T09:44:54.125ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/240cabb9-b684-4172-8606-d31cd2279cd6/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:879
23922024-05-14T09:44:54.263ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/240cabb9-b684-4172-8606-d31cd2279cd6/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:879
23932024-05-14T09:44:54.398ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/240cabb9-b684-4172-8606-d31cd2279cd6/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:879
23942024-05-14T09:44:54.734ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/240cabb9-b684-4172-8606-d31cd2279cd6/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:879
23952024-05-14T09:44:54.966ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/240cabb9-b684-4172-8606-d31cd2279cd6/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:879
23962024-05-14T09:44:55.152ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/240cabb9-b684-4172-8606-d31cd2279cd6/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:879
23972024-05-14T09:44:55.513ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/240cabb9-b684-4172-8606-d31cd2279cd6/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:879
23982024-05-14T09:44:56.171ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/240cabb9-b684-4172-8606-d31cd2279cd6/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:879
23992024-05-14T09:44:56.987ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/240cabb9-b684-4172-8606-d31cd2279cd6/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:879
24002024-05-14T09:44:57.851ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/240cabb9-b684-4172-8606-d31cd2279cd6/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:879
24012024-05-14T09:44:58.763ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/240cabb9-b684-4172-8606-d31cd2279cd6/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:879
24022024-05-14T09:44:59.362ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/240cabb9-b684-4172-8606-d31cd2279cd6/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:879
24032024-05-14T09:44:59.378ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/5f617488-2ef8-4cf1-81f7-2b249f5b34c0): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:619 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
24042024-05-14T09:44:59.608ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/5f617488-2ef8-4cf1-81f7-2b249f5b34c0): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:169 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
24052024-05-14T09:45:00.614ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/5f617488-2ef8-4cf1-81f7-2b249f5b34c0): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:169 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
24062024-05-14T09:45:00.803ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/5f617488-2ef8-4cf1-81f7-2b249f5b34c0): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:619 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
24072024-05-14T09:45:00.948ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:42681
24082024-05-14T09:45:00.949ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 40 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:42681 req_id = 05c420b5-abc5-4d13-a4ed-575c5159b0b1 response_code = 200 uri = /baseboard
24092024-05-14T09:45:01.268ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/240cabb9-b684-4172-8606-d31cd2279cd6/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:879
24102024-05-14T09:45:01.618ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/5f617488-2ef8-4cf1-81f7-2b249f5b34c0): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:169 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
24112024-05-14T09:45:02.549ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/240cabb9-b684-4172-8606-d31cd2279cd6/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:879
24122024-05-14T09:45:02.622ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/5f617488-2ef8-4cf1-81f7-2b249f5b34c0): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:169 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
24132024-05-14T09:45:02.804ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/5f617488-2ef8-4cf1-81f7-2b249f5b34c0): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:619 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
24142024-05-14T09:45:03.817ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "1623130b-4b07-46da-a08d-a1e6e21f1ae8", "content-length": "173", "date": "Tue, 14 May 2024 09:45:03 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: rack with id \"240cabb9-b684-4172-8606-d31cd2279cd6\"", request_id: "1623130b-4b07-46da-a08d-a1e6e21f1ae8" } file = sled-agent/src/rack_setup/service.rs:879
24152024-05-14T09:45:06.933ZINFOSledAgent (NexusNotifierTask): Successfully put SledAgentInfo to nexus file = sled-agent/src/nexus.rs:592 new_generation = 1 old_generation = 1 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
24162024-05-14T09:45:07.166ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:42055
24172024-05-14T09:45:07.167ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
24182024-05-14T09:45:07.265ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 98884 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:42055 req_id = 37322965-5213-4322-b1f7-b450c2d0ff13 response_code = 200 uri = /inventory
24192024-05-14T09:45:07.267ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
24202024-05-14T09:45:07.267ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
24212024-05-14T09:45:07.275ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
24222024-05-14T09:45:07.275ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 9473 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:42055 req_id = 939f2b38-806b-4c77-90d2-df1f6aa71920 response_code = 200 uri = /omicron-zones
24232024-05-14T09:45:07.830ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:54178
24242024-05-14T09:45:07.834ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:525 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: b5cecdd2-5810-47a4-acef-30ed4d68f165, kind: Service { id: 7d57f350-6d84-441e-aca9-427498419506 }, name: Name("external-dns-7d57f350-6d84-441e-aca9-427498419506"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 209, 89])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: c3a0d6a8-690b-4d68-9c53-ea94903a313d, kind: Service { id: bd856afa-64af-4f62-8083-7cec6f530e9b }, name: Name("external-dns-bd856afa-64af-4f62-8083-7cec6f530e9b"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 238, 230])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0e69104d-a1e3-43ae-9fa7-d0ba3efaba6d, kind: Service { id: d7310b7c-67e1-4338-8414-09460b7646e9 }, name: Name("nexus-d7310b7c-67e1-4338-8414-09460b7646e9"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 218, 154])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: afa350c1-4cae-457d-b6fa-02b99bddeae1, kind: Service { id: 10cdd4df-0856-4bcf-af87-e5757529b0c3 }, name: Name("nexus-10cdd4df-0856-4bcf-af87-e5757529b0c3"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 145, 29])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: b34dcba8-9ce7-4e76-8638-6efaa0e9a52e, kind: Service { id: 1af27ceb-2d94-42b4-abb9-e7054fec8ade }, name: Name("nexus-1af27ceb-2d94-42b4-abb9-e7054fec8ade"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 214, 74])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
24252024-05-14T09:45:07.834ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
24262024-05-14T09:45:07.834ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
24272024-05-14T09:45:07.834ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
24282024-05-14T09:45:07.834ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
24292024-05-14T09:45:07.834ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
24302024-05-14T09:45:07.834ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte0" rules = []
24312024-05-14T09:45:07.834ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 4577 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:54178 req_id = f755b494-f47c-42db-b204-24124d82a89e response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
24322024-05-14T09:45:07.837ZINFOSledAgent (RSS): Handoff to Nexus is complete file = sled-agent/src/rack_setup/service.rs:889
24332024-05-14T09:45:08.045ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:62493
24342024-05-14T09:45:08.045ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
24352024-05-14T09:45:08.139ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 93821 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:62493 req_id = 5bc69e61-9db6-4e61-893b-e3734958b41c response_code = 200 uri = /inventory
24362024-05-14T09:45:08.139ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
24372024-05-14T09:45:08.139ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/112d7827-193f-456c-94a5-19163d29475e/config/omicron-zones.json file = common/src/ledger.rs:177
24382024-05-14T09:45:08.140ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/config/omicron-zones.json file = common/src/ledger.rs:177
24392024-05-14T09:45:08.140ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 1450 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:62493 req_id = 8b001f7e-d75e-49d3-9fdd-4cb41660e0b7 response_code = 200 uri = /omicron-zones
24402024-05-14T09:45:09.711ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37099
24412024-05-14T09:45:09.711ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:525 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: b5cecdd2-5810-47a4-acef-30ed4d68f165, kind: Service { id: 7d57f350-6d84-441e-aca9-427498419506 }, name: Name("external-dns-7d57f350-6d84-441e-aca9-427498419506"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 209, 89])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: c3a0d6a8-690b-4d68-9c53-ea94903a313d, kind: Service { id: bd856afa-64af-4f62-8083-7cec6f530e9b }, name: Name("external-dns-bd856afa-64af-4f62-8083-7cec6f530e9b"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 238, 230])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0e69104d-a1e3-43ae-9fa7-d0ba3efaba6d, kind: Service { id: d7310b7c-67e1-4338-8414-09460b7646e9 }, name: Name("nexus-d7310b7c-67e1-4338-8414-09460b7646e9"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 218, 154])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: afa350c1-4cae-457d-b6fa-02b99bddeae1, kind: Service { id: 10cdd4df-0856-4bcf-af87-e5757529b0c3 }, name: Name("nexus-10cdd4df-0856-4bcf-af87-e5757529b0c3"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 145, 29])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: b34dcba8-9ce7-4e76-8638-6efaa0e9a52e, kind: Service { id: 1af27ceb-2d94-42b4-abb9-e7054fec8ade }, name: Name("nexus-1af27ceb-2d94-42b4-abb9-e7054fec8ade"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 214, 74])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
24422024-05-14T09:45:09.711ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
24432024-05-14T09:45:09.711ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
24442024-05-14T09:45:09.711ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
24452024-05-14T09:45:09.711ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
24462024-05-14T09:45:09.711ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
24472024-05-14T09:45:09.711ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte0" rules = []
24482024-05-14T09:45:09.711ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 628 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:37099 req_id = 21f2c083-7138-47b5-9c14-4ba24dd8bbf6 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
24492024-05-14T09:45:09.723ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:60138
24502024-05-14T09:45:09.734ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:525 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: b5cecdd2-5810-47a4-acef-30ed4d68f165, kind: Service { id: 7d57f350-6d84-441e-aca9-427498419506 }, name: Name("external-dns-7d57f350-6d84-441e-aca9-427498419506"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 209, 89])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: c3a0d6a8-690b-4d68-9c53-ea94903a313d, kind: Service { id: bd856afa-64af-4f62-8083-7cec6f530e9b }, name: Name("external-dns-bd856afa-64af-4f62-8083-7cec6f530e9b"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 238, 230])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0e69104d-a1e3-43ae-9fa7-d0ba3efaba6d, kind: Service { id: d7310b7c-67e1-4338-8414-09460b7646e9 }, name: Name("nexus-d7310b7c-67e1-4338-8414-09460b7646e9"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 218, 154])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: afa350c1-4cae-457d-b6fa-02b99bddeae1, kind: Service { id: 10cdd4df-0856-4bcf-af87-e5757529b0c3 }, name: Name("nexus-10cdd4df-0856-4bcf-af87-e5757529b0c3"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 145, 29])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: b34dcba8-9ce7-4e76-8638-6efaa0e9a52e, kind: Service { id: 1af27ceb-2d94-42b4-abb9-e7054fec8ade }, name: Name("nexus-1af27ceb-2d94-42b4-abb9-e7054fec8ade"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 214, 74])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
24512024-05-14T09:45:09.734ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
24522024-05-14T09:45:09.738ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
24532024-05-14T09:45:09.738ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
24542024-05-14T09:45:09.738ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
24552024-05-14T09:45:09.738ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
24562024-05-14T09:45:09.738ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte0" rules = []
24572024-05-14T09:45:09.738ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 14961 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:60138 req_id = 221c2289-c449-4391-accc-357f36b39cac response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
24582024-05-14T09:45:09.800ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:56117
24592024-05-14T09:45:09.801ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:525 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: b5cecdd2-5810-47a4-acef-30ed4d68f165, kind: Service { id: 7d57f350-6d84-441e-aca9-427498419506 }, name: Name("external-dns-7d57f350-6d84-441e-aca9-427498419506"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 209, 89])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: c3a0d6a8-690b-4d68-9c53-ea94903a313d, kind: Service { id: bd856afa-64af-4f62-8083-7cec6f530e9b }, name: Name("external-dns-bd856afa-64af-4f62-8083-7cec6f530e9b"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 238, 230])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0e69104d-a1e3-43ae-9fa7-d0ba3efaba6d, kind: Service { id: d7310b7c-67e1-4338-8414-09460b7646e9 }, name: Name("nexus-d7310b7c-67e1-4338-8414-09460b7646e9"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 218, 154])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: afa350c1-4cae-457d-b6fa-02b99bddeae1, kind: Service { id: 10cdd4df-0856-4bcf-af87-e5757529b0c3 }, name: Name("nexus-10cdd4df-0856-4bcf-af87-e5757529b0c3"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 145, 29])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: b34dcba8-9ce7-4e76-8638-6efaa0e9a52e, kind: Service { id: 1af27ceb-2d94-42b4-abb9-e7054fec8ade }, name: Name("nexus-1af27ceb-2d94-42b4-abb9-e7054fec8ade"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 214, 74])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
24602024-05-14T09:45:09.801ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
24612024-05-14T09:45:09.801ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
24622024-05-14T09:45:09.801ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
24632024-05-14T09:45:09.801ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
24642024-05-14T09:45:09.801ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
24652024-05-14T09:45:09.801ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte0" rules = []
24662024-05-14T09:45:09.801ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 388 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:56117 req_id = f3724341-ae62-4478-acb9-ea9be7c4e945 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
24672024-05-14T09:45:30.987ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:47203
24682024-05-14T09:45:30.988ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 59 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:47203 req_id = 89d62437-643e-4797-a6bf-bccd4b7b358b response_code = 200 uri = /baseboard
24692024-05-14T09:45:33.753ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59166
24702024-05-14T09:45:33.834ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 80395 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:59166 req_id = 67bcaf5f-5d47-492b-9b25-63ceab0bbaf0 response_code = 204 uri = /switch-ports
24712024-05-14T09:45:33.834ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36118
24722024-05-14T09:45:33.834ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:60078
24732024-05-14T09:45:33.839ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 4430 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:59166 req_id = a5db7146-0ddb-4f50-a7bb-b77483374119 response_code = 200 uri = /network-bootstore-config
24742024-05-14T09:45:33.860ZINFOSledAgent (bootstore): Attempting to update network config with generation: 2, current_generation: 1 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-sock-i86pc
24752024-05-14T09:45:33.860ZINFOSledAgent (bootstore): Writing ledger to /pool/int/112d7827-193f-456c-94a5-19163d29475e/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
24762024-05-14T09:45:33.861ZINFOSledAgent (bootstore): Writing ledger to /pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
24772024-05-14T09:45:33.862ZINFOSledAgent (bootstore): Broadcasting network config with generation 2 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-sock-i86pc
24782024-05-14T09:45:33.862ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 2826 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:59166 req_id = bf6da70a-c81a-4a0b-af1e-cf5e6c314f14 response_code = 204 uri = /network-bootstore-config
24792024-05-14T09:45:33.891ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 56837 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:36118 req_id = e2934d8c-c75c-46be-bc8e-09e42a47c661 response_code = 204 uri = /switch-ports
24802024-05-14T09:45:33.895ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 1059 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:36118 req_id = 9eb4286a-a6ed-4685-8cc4-61cc837fc6ee response_code = 200 uri = /network-bootstore-config
24812024-05-14T09:45:33.942ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 108287 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:60078 req_id = 0342e2e4-4342-4413-a003-2bfa06cd2081 response_code = 204 uri = /switch-ports
24822024-05-14T09:45:33.943ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 111 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:60078 req_id = 39e99e76-6cc4-4e44-a177-2201eef06566 response_code = 200 uri = /network-bootstore-config
24832024-05-14T09:46:01.020ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:44202
24842024-05-14T09:46:01.020ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 82 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:44202 req_id = ee01c67a-6838-46dc-825a-0a747288bcd9 response_code = 200 uri = /baseboard
24852024-05-14T09:46:03.821ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:44323
24862024-05-14T09:46:03.875ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 53512 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:44323 req_id = ae6af296-291f-40b3-9ab3-524052e07287 response_code = 204 uri = /switch-ports
24872024-05-14T09:46:03.875ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 89 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:44323 req_id = 6d961d2b-516c-4507-9984-847cdbd7e7d7 response_code = 200 uri = /network-bootstore-config
24882024-05-14T09:46:03.927ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47867
24892024-05-14T09:46:03.979ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 51381 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:47867 req_id = 9c1ba0f5-8b67-44cf-b403-721d2f90d2d9 response_code = 204 uri = /switch-ports
24902024-05-14T09:46:03.980ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 91 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47867 req_id = 919bba37-4485-4bd7-a086-5a4006944b56 response_code = 200 uri = /network-bootstore-config
24912024-05-14T09:46:04.010ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:43058
24922024-05-14T09:46:04.060ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 49715 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:43058 req_id = dc2bed7a-44bb-43a9-a7ac-ea9d953f6bab response_code = 204 uri = /switch-ports
24932024-05-14T09:46:04.060ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 116 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:43058 req_id = c86c87a7-7bfc-441f-8f15-69ddc6787958 response_code = 200 uri = /network-bootstore-config
24942024-05-14T09:46:31.051ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:58546
24952024-05-14T09:46:31.051ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 96 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:58546 req_id = 5000662e-23b6-4426-9232-ae243d1a32a7 response_code = 200 uri = /baseboard
24962024-05-14T09:46:33.622ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:64085
24972024-05-14T09:46:33.671ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 48305 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:64085 req_id = 6bec7ae2-d6f8-4303-b8f2-5b99b1789764 response_code = 204 uri = /switch-ports
24982024-05-14T09:46:33.671ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:48017
24992024-05-14T09:46:33.675ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 3594 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:64085 req_id = b6fd541a-fa4c-4976-b4bb-5f45b9ab8e9a response_code = 200 uri = /network-bootstore-config
25002024-05-14T09:46:33.717ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 46148 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:48017 req_id = 5d0e5246-d9e8-406a-bfdc-7f4805b270f1 response_code = 204 uri = /switch-ports
25012024-05-14T09:46:33.718ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 165 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:48017 req_id = 937505bc-1d58-4ecf-95ab-29b7ba2864f9 response_code = 200 uri = /network-bootstore-config
25022024-05-14T09:46:33.735ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:60760
25032024-05-14T09:46:33.779ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 44458 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:60760 req_id = 5d49a082-8940-4709-a52a-5ec0ebe75668 response_code = 204 uri = /switch-ports
25042024-05-14T09:46:33.780ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 144 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:60760 req_id = f8bf819b-e316-416a-bc03-235460d2a022 response_code = 200 uri = /network-bootstore-config
25052024-05-14T09:46:50.903ZINFOSledAgent (dropshot): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:51153 remote_addr = [fd00:1122:3344:101::d]:45846 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0
25062024-05-14T09:46:50.916ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 12432 local_addr = [fd00:1122:3344:101::1]:51153 method = GET remote_addr = [fd00:1122:3344:101::d]:45846 req_id = 82a7380e-d926-4e0f-a70c-d844b3b1f745 response_code = 200 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0 uri = //5f617488-2ef8-4cf1-81f7-2b249f5b34c0
25072024-05-14T09:47:01.081ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:40398
25082024-05-14T09:47:01.081ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 85 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:40398 req_id = b7067b5c-d838-4c4d-83fc-577225b3a53e response_code = 200 uri = /baseboard
25092024-05-14T09:47:03.855ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:43079
25102024-05-14T09:47:03.907ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 52734 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:43079 req_id = fa778799-330a-417c-8efa-1edb7e70882d response_code = 204 uri = /switch-ports
25112024-05-14T09:47:03.908ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44993
25122024-05-14T09:47:03.958ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 50491 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:44993 req_id = 15fbcea3-9d18-4816-b925-e34c75b22ec7 response_code = 204 uri = /switch-ports
25132024-05-14T09:47:03.959ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 114 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44993 req_id = 25b52e41-44b2-4084-a760-25c85803fcb5 response_code = 200 uri = /network-bootstore-config
25142024-05-14T09:47:03.959ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 213 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:43079 req_id = 0ab126cf-8117-491b-9e30-e9e3d3c268dd response_code = 200 uri = /network-bootstore-config
25152024-05-14T09:47:04.138ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:44751
25162024-05-14T09:47:04.183ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 44440 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:44751 req_id = 71395211-6beb-4b68-b67e-b7f040d57924 response_code = 204 uri = /switch-ports
25172024-05-14T09:47:04.184ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 877 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:44751 req_id = 1847ae36-e671-4f66-9e15-e8cab537a907 response_code = 200 uri = /network-bootstore-config
25182024-05-14T09:47:20.903ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 684 local_addr = [fd00:1122:3344:101::1]:51153 method = GET remote_addr = [fd00:1122:3344:101::d]:45846 req_id = f5d203b2-9465-4532-b62e-745b8fc8f4f4 response_code = 200 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0 uri = //5f617488-2ef8-4cf1-81f7-2b249f5b34c0
25192024-05-14T09:47:31.115ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:33289
25202024-05-14T09:47:31.115ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 86 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:33289 req_id = 9a95376b-ec4f-4b47-8f64-e69fbd2b1dc3 response_code = 200 uri = /baseboard
25212024-05-14T09:47:33.635ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40004
25222024-05-14T09:47:33.685ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 49575 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:40004 req_id = 6d358129-19f6-4eab-a999-37d5e3f69313 response_code = 204 uri = /switch-ports
25232024-05-14T09:47:33.685ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:54965
25242024-05-14T09:47:33.688ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 3264 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40004 req_id = eb1da26a-075d-4960-94ee-108b90cc07a6 response_code = 200 uri = /network-bootstore-config
25252024-05-14T09:47:33.731ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 46056 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:54965 req_id = 7b3919f5-5e61-40e3-9ac7-57699da534a9 response_code = 204 uri = /switch-ports
25262024-05-14T09:47:33.732ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 853 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:54965 req_id = 7498ac9d-b278-4f49-b9ca-6bc924eccf38 response_code = 200 uri = /network-bootstore-config
25272024-05-14T09:47:33.738ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:39734
25282024-05-14T09:47:33.783ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 45150 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:39734 req_id = c8ceaa92-ea98-4aca-ad7e-f0648ea41f22 response_code = 204 uri = /switch-ports
25292024-05-14T09:47:33.783ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 98 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:39734 req_id = e58824f7-8d67-4aeb-b8d3-7519ea5b1a97 response_code = 200 uri = /network-bootstore-config
25302024-05-14T09:47:50.903ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 533 local_addr = [fd00:1122:3344:101::1]:51153 method = GET remote_addr = [fd00:1122:3344:101::d]:45846 req_id = 58e4a3aa-c30b-4821-b774-5c4b00d2f24e response_code = 200 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0 uri = //5f617488-2ef8-4cf1-81f7-2b249f5b34c0
25312024-05-14T09:48:01.148ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:65240
25322024-05-14T09:48:01.148ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 65 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:65240 req_id = d5df1c95-1cba-4b73-a577-780c24afdef0 response_code = 200 uri = /baseboard
25332024-05-14T09:48:03.754ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:61755
25342024-05-14T09:48:03.805ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 51252 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:61755 req_id = a780facc-c34d-4dd4-8910-6f2dd093436a response_code = 204 uri = /switch-ports
25352024-05-14T09:48:03.806ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 98 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:61755 req_id = 46c3bcfc-e1bf-4467-8f3a-c940f8813691 response_code = 200 uri = /network-bootstore-config
25362024-05-14T09:48:03.844ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:57322
25372024-05-14T09:48:03.895ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 51383 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:57322 req_id = c86ce060-9883-4ba6-826a-079a119df351 response_code = 204 uri = /switch-ports
25382024-05-14T09:48:03.895ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 104 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:57322 req_id = fadb1009-bdf8-4841-a940-99d4caafbfc0 response_code = 200 uri = /network-bootstore-config
25392024-05-14T09:48:03.910ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:35703
25402024-05-14T09:48:03.957ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 46762 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:35703 req_id = 0213566c-cf81-4b93-866f-150bcbb6a313 response_code = 204 uri = /switch-ports
25412024-05-14T09:48:03.957ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 96 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:35703 req_id = 380b1e9f-8829-49d1-93da-3b712664f74c response_code = 200 uri = /network-bootstore-config
25422024-05-14T09:48:20.903ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 1051 local_addr = [fd00:1122:3344:101::1]:51153 method = GET remote_addr = [fd00:1122:3344:101::d]:45846 req_id = 366fcda4-fc11-4043-9696-4cc5f4da30a7 response_code = 200 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0 uri = //5f617488-2ef8-4cf1-81f7-2b249f5b34c0
25432024-05-14T09:48:31.180ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:52808
25442024-05-14T09:48:31.180ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 99 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:52808 req_id = 0bfb93fd-33c2-46fb-9214-58c660028b16 response_code = 200 uri = /baseboard
25452024-05-14T09:48:33.634ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:43099
25462024-05-14T09:48:33.686ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 51478 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:43099 req_id = a90c3b94-66bb-41f8-9ca7-5f5d57bd7ae2 response_code = 204 uri = /switch-ports
25472024-05-14T09:48:33.686ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:35225
25482024-05-14T09:48:33.690ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 4078 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:43099 req_id = 30a165d3-228c-4dd8-a372-55de376fa4c2 response_code = 200 uri = /network-bootstore-config
25492024-05-14T09:48:33.732ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 46027 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:35225 req_id = 33ade52b-f00c-405c-bcd6-5051f2e21b9f response_code = 204 uri = /switch-ports
25502024-05-14T09:48:33.732ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 130 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:35225 req_id = 625409ce-3cd1-46e9-9957-2c112788c275 response_code = 200 uri = /network-bootstore-config
25512024-05-14T09:48:33.802ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:53321
25522024-05-14T09:48:33.847ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 44642 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:53321 req_id = 610bf0e9-13cb-4bd0-8f8e-398e4f5b42df response_code = 204 uri = /switch-ports
25532024-05-14T09:48:33.847ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 91 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:53321 req_id = 9da99f29-777d-4d28-ad10-abcfaea8cb78 response_code = 200 uri = /network-bootstore-config
25542024-05-14T09:48:50.908ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 1361 local_addr = [fd00:1122:3344:101::1]:51153 method = GET remote_addr = [fd00:1122:3344:101::d]:45846 req_id = 4304391d-7264-4be5-87d6-adc3ad942f04 response_code = 200 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0 uri = //5f617488-2ef8-4cf1-81f7-2b249f5b34c0
25552024-05-14T09:49:01.211ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:51006
25562024-05-14T09:49:01.211ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 89 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:51006 req_id = 47138253-8e82-4960-a437-ec669e5e3de4 response_code = 200 uri = /baseboard
25572024-05-14T09:49:03.826ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:56800
25582024-05-14T09:49:03.826ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:48420
25592024-05-14T09:49:03.872ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 46016 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:56800 req_id = c8c45af8-355c-4b19-b5bf-4fbca8537cbc response_code = 204 uri = /switch-ports
25602024-05-14T09:49:03.876ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 440 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:56800 req_id = 7466514e-f792-470c-81c8-f8c378f4fdc7 response_code = 200 uri = /network-bootstore-config
25612024-05-14T09:49:03.920ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 93660 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:48420 req_id = 4ff948d3-957f-41f6-b097-90831ac620f7 response_code = 204 uri = /switch-ports
25622024-05-14T09:49:03.921ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 142 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:48420 req_id = b3d91cfa-c73a-40ae-a518-c3838e4e6271 response_code = 200 uri = /network-bootstore-config
25632024-05-14T09:49:04.065ZINFOSledAgent (auto-cleanup-task): running automatic periodic zone bundle cleanup file = sled-agent/src/zone_bundle.rs:300
25642024-05-14T09:49:04.066ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
25652024-05-14T09:49:04.106ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:289 next_cleanup = Instant { tv_sec: 1953, tv_nsec: 481619869 } time_to_next_cleanup = 599.99999646s
25662024-05-14T09:49:04.185ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52553
25672024-05-14T09:49:04.233ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 48770 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:52553 req_id = 4ee94650-b6da-45e5-bc58-847330e9f705 response_code = 204 uri = /switch-ports
25682024-05-14T09:49:04.234ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 118 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:52553 req_id = e6f36dd7-5293-4bdd-8c76-af34ac909500 response_code = 200 uri = /network-bootstore-config
25692024-05-14T09:49:20.945ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 1288 local_addr = [fd00:1122:3344:101::1]:51153 method = GET remote_addr = [fd00:1122:3344:101::d]:45846 req_id = 08b643e8-7213-4088-b31d-a3b79e07cb21 response_code = 200 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0 uri = //5f617488-2ef8-4cf1-81f7-2b249f5b34c0
25702024-05-14T09:49:31.244ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:50292
25712024-05-14T09:49:31.244ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 101 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:50292 req_id = 31691f0d-1698-4241-aff1-e553da38cfbd response_code = 200 uri = /baseboard
25722024-05-14T09:49:33.725ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:59729
25732024-05-14T09:49:33.775ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 49820 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:59729 req_id = ac8ea729-4bdb-4869-9086-4ddd114f6656 response_code = 204 uri = /switch-ports
25742024-05-14T09:49:33.775ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 137 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:59729 req_id = 2eab15b1-5ba5-4e97-aad1-492dce8880de response_code = 200 uri = /network-bootstore-config
25752024-05-14T09:49:33.776ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:58361
25762024-05-14T09:49:33.820ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 44330 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:58361 req_id = 0e8e3050-a489-4f43-912c-3866e0faf33e response_code = 204 uri = /switch-ports
25772024-05-14T09:49:33.820ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 124 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:58361 req_id = 670b8771-bfa5-4b2d-85c4-0d8f962f5e58 response_code = 200 uri = /network-bootstore-config
25782024-05-14T09:49:33.838ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:62340
25792024-05-14T09:49:33.883ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 44165 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:62340 req_id = 2888f17a-9b36-4084-9454-3dc71a3ca5f3 response_code = 204 uri = /switch-ports
25802024-05-14T09:49:33.883ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:62340 req_id = a92be0c8-3e77-45ab-984e-6b68d44140a6 response_code = 200 uri = /network-bootstore-config
25812024-05-14T09:49:47.285ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:801
25822024-05-14T09:49:50.944ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 703 local_addr = [fd00:1122:3344:101::1]:51153 method = GET remote_addr = [fd00:1122:3344:101::d]:45846 req_id = ebdbce55-1c2d-4f34-aa79-7ce3d523cb76 response_code = 200 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0 uri = //5f617488-2ef8-4cf1-81f7-2b249f5b34c0
25832024-05-14T09:50:01.276ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:41002
25842024-05-14T09:50:01.276ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 121 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:41002 req_id = 845cfead-8a99-4e0c-8424-d96ac786e375 response_code = 200 uri = /baseboard
25852024-05-14T09:50:03.847ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:39926
25862024-05-14T09:50:03.896ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 48544 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:39926 req_id = 2c874986-51c1-429f-8b0a-bf94aa73dd83 response_code = 204 uri = /switch-ports
25872024-05-14T09:50:03.896ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:42055
25882024-05-14T09:50:03.896ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:525 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: b5cecdd2-5810-47a4-acef-30ed4d68f165, kind: Service { id: 7d57f350-6d84-441e-aca9-427498419506 }, name: Name("external-dns-7d57f350-6d84-441e-aca9-427498419506"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 209, 89])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: c3a0d6a8-690b-4d68-9c53-ea94903a313d, kind: Service { id: bd856afa-64af-4f62-8083-7cec6f530e9b }, name: Name("external-dns-bd856afa-64af-4f62-8083-7cec6f530e9b"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 238, 230])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0e69104d-a1e3-43ae-9fa7-d0ba3efaba6d, kind: Service { id: d7310b7c-67e1-4338-8414-09460b7646e9 }, name: Name("nexus-d7310b7c-67e1-4338-8414-09460b7646e9"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 218, 154])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: afa350c1-4cae-457d-b6fa-02b99bddeae1, kind: Service { id: 10cdd4df-0856-4bcf-af87-e5757529b0c3 }, name: Name("nexus-10cdd4df-0856-4bcf-af87-e5757529b0c3"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 145, 29])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: b34dcba8-9ce7-4e76-8638-6efaa0e9a52e, kind: Service { id: 1af27ceb-2d94-42b4-abb9-e7054fec8ade }, name: Name("nexus-1af27ceb-2d94-42b4-abb9-e7054fec8ade"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 214, 74])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
25892024-05-14T09:50:03.896ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 190 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:39926 req_id = f7d7fd7e-f68b-4ef4-80ca-db2e397d1628 response_code = 200 uri = /network-bootstore-config
25902024-05-14T09:50:03.896ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
25912024-05-14T09:50:03.896ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
25922024-05-14T09:50:03.896ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
25932024-05-14T09:50:03.896ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
25942024-05-14T09:50:03.896ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
25952024-05-14T09:50:03.896ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte0" rules = []
25962024-05-14T09:50:03.896ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 386 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:42055 req_id = c1789bd2-e8f4-45e8-b779-190e42a5076e response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
25972024-05-14T09:50:04.017ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:52937
25982024-05-14T09:50:04.017ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:525 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: b5cecdd2-5810-47a4-acef-30ed4d68f165, kind: Service { id: 7d57f350-6d84-441e-aca9-427498419506 }, name: Name("external-dns-7d57f350-6d84-441e-aca9-427498419506"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 209, 89])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: c3a0d6a8-690b-4d68-9c53-ea94903a313d, kind: Service { id: bd856afa-64af-4f62-8083-7cec6f530e9b }, name: Name("external-dns-bd856afa-64af-4f62-8083-7cec6f530e9b"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 238, 230])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0e69104d-a1e3-43ae-9fa7-d0ba3efaba6d, kind: Service { id: d7310b7c-67e1-4338-8414-09460b7646e9 }, name: Name("nexus-d7310b7c-67e1-4338-8414-09460b7646e9"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 218, 154])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: afa350c1-4cae-457d-b6fa-02b99bddeae1, kind: Service { id: 10cdd4df-0856-4bcf-af87-e5757529b0c3 }, name: Name("nexus-10cdd4df-0856-4bcf-af87-e5757529b0c3"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 145, 29])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: b34dcba8-9ce7-4e76-8638-6efaa0e9a52e, kind: Service { id: 1af27ceb-2d94-42b4-abb9-e7054fec8ade }, name: Name("nexus-1af27ceb-2d94-42b4-abb9-e7054fec8ade"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 214, 74])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
25992024-05-14T09:50:04.017ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26002024-05-14T09:50:04.018ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26012024-05-14T09:50:04.018ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26022024-05-14T09:50:04.018ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
26032024-05-14T09:50:04.018ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
26042024-05-14T09:50:04.018ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte0" rules = []
26052024-05-14T09:50:04.018ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 392 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:52937 req_id = e350ca37-7cc2-4351-81a3-fc110c95339b response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
26062024-05-14T09:50:04.055ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:64534
26072024-05-14T09:50:04.106ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 50826 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:64534 req_id = 56ac1dd1-5b20-41b9-9a5b-f6ab7cdbce4e response_code = 204 uri = /switch-ports
26082024-05-14T09:50:04.106ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:49191
26092024-05-14T09:50:04.107ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:43360
26102024-05-14T09:50:04.107ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:525 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: b5cecdd2-5810-47a4-acef-30ed4d68f165, kind: Service { id: 7d57f350-6d84-441e-aca9-427498419506 }, name: Name("external-dns-7d57f350-6d84-441e-aca9-427498419506"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 209, 89])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: c3a0d6a8-690b-4d68-9c53-ea94903a313d, kind: Service { id: bd856afa-64af-4f62-8083-7cec6f530e9b }, name: Name("external-dns-bd856afa-64af-4f62-8083-7cec6f530e9b"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 238, 230])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0e69104d-a1e3-43ae-9fa7-d0ba3efaba6d, kind: Service { id: d7310b7c-67e1-4338-8414-09460b7646e9 }, name: Name("nexus-d7310b7c-67e1-4338-8414-09460b7646e9"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 218, 154])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: afa350c1-4cae-457d-b6fa-02b99bddeae1, kind: Service { id: 10cdd4df-0856-4bcf-af87-e5757529b0c3 }, name: Name("nexus-10cdd4df-0856-4bcf-af87-e5757529b0c3"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 145, 29])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: b34dcba8-9ce7-4e76-8638-6efaa0e9a52e, kind: Service { id: 1af27ceb-2d94-42b4-abb9-e7054fec8ade }, name: Name("nexus-1af27ceb-2d94-42b4-abb9-e7054fec8ade"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 214, 74])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
26112024-05-14T09:50:04.107ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 235 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:64534 req_id = 73658f07-83d4-4067-8eb2-92b8480f2c10 response_code = 200 uri = /network-bootstore-config
26122024-05-14T09:50:04.107ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26132024-05-14T09:50:04.107ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26142024-05-14T09:50:04.111ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26152024-05-14T09:50:04.111ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
26162024-05-14T09:50:04.111ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
26172024-05-14T09:50:04.111ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte0" rules = []
26182024-05-14T09:50:04.111ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 4577 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:49191 req_id = 19af518e-8437-4e53-a018-180cc10a6cf5 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
26192024-05-14T09:50:04.157ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 49960 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:43360 req_id = 00c1352d-b349-44be-9902-6f134d62fa88 response_code = 204 uri = /switch-ports
26202024-05-14T09:50:04.157ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 197 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:43360 req_id = 8e5c7462-867e-4762-ad05-73300da99fe4 response_code = 200 uri = /network-bootstore-config
26212024-05-14T09:50:20.944ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 570 local_addr = [fd00:1122:3344:101::1]:51153 method = GET remote_addr = [fd00:1122:3344:101::d]:45846 req_id = 3a69b1d7-b18c-4e64-9f6f-cbe45ce0667d response_code = 200 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0 uri = //5f617488-2ef8-4cf1-81f7-2b249f5b34c0
26222024-05-14T09:50:31.308ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:62772
26232024-05-14T09:50:31.308ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 72 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:62772 req_id = 8dfbd103-e9ea-47ce-bc6f-8693a7f9acee response_code = 200 uri = /baseboard
26242024-05-14T09:50:33.643ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:38790
26252024-05-14T09:50:33.692ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 48700 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:38790 req_id = 58d16d0e-3029-40ce-97d6-a4aac9a6166d response_code = 204 uri = /switch-ports
26262024-05-14T09:50:33.692ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:41238
26272024-05-14T09:50:33.738ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 46433 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:41238 req_id = 158e8328-7252-4902-b408-49cdf9fac4bd response_code = 204 uri = /switch-ports
26282024-05-14T09:50:33.738ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:55651
26292024-05-14T09:50:33.742ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 3399 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:38790 req_id = 7ac311fa-6a54-45a0-8a24-933b6164d448 response_code = 200 uri = /network-bootstore-config
26302024-05-14T09:50:33.742ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 3440 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:41238 req_id = a9783d7b-6572-48a1-ba50-f7e2cc7f01a2 response_code = 200 uri = /network-bootstore-config
26312024-05-14T09:50:33.787ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 48118 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:55651 req_id = 5c541243-a862-41f3-ab38-7b7c82a02ee3 response_code = 204 uri = /switch-ports
26322024-05-14T09:50:33.787ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 160 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:55651 req_id = 9a9ea983-df0a-45b2-9256-a7fc01d82629 response_code = 200 uri = /network-bootstore-config
26332024-05-14T09:50:50.943ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 553 local_addr = [fd00:1122:3344:101::1]:51153 method = GET remote_addr = [fd00:1122:3344:101::d]:45846 req_id = bbabeb2a-7cb7-4567-bd58-02f488900c7c response_code = 200 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0 uri = //5f617488-2ef8-4cf1-81f7-2b249f5b34c0
26342024-05-14T09:51:01.339ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:39753
26352024-05-14T09:51:01.339ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 63 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:39753 req_id = 9894e76e-8f67-4323-b5dc-c68be036c4b9 response_code = 200 uri = /baseboard
26362024-05-14T09:51:03.836ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50161
26372024-05-14T09:51:03.890ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 53689 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:50161 req_id = 932da1a2-0bc4-44c0-ac39-a154293b8f7a response_code = 204 uri = /switch-ports
26382024-05-14T09:51:03.891ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 900 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50161 req_id = 135a3479-fa0b-4dfa-b2e5-481665706757 response_code = 200 uri = /network-bootstore-config
26392024-05-14T09:51:03.908ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:60390
26402024-05-14T09:51:03.957ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 49447 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:60390 req_id = f3beffdf-9e11-4bd4-a3de-7aa2248c8006 response_code = 204 uri = /switch-ports
26412024-05-14T09:51:03.957ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:53597
26422024-05-14T09:51:03.961ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 3476 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:60390 req_id = 21d96105-a934-48bd-a80f-b63e9ad1956e response_code = 200 uri = /network-bootstore-config
26432024-05-14T09:51:04.004ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 46678 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:53597 req_id = 95bc9226-02c7-44aa-93d2-1cd9cd1e74df response_code = 204 uri = /switch-ports
26442024-05-14T09:51:04.004ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 131 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:53597 req_id = 67790081-02fe-46ff-a67c-e970fac5f939 response_code = 200 uri = /network-bootstore-config
26452024-05-14T09:51:10.262ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:32960
26462024-05-14T09:51:10.278ZINFOSledAgent (InstanceManager): ensuring instance is registered file = sled-agent/src/instance_manager.rs:546 hardware = InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: 49ed10a6-8516-40ff-a573-5f419cb03dfc, kind: Instance { id: 1f8794d5-895b-43ae-93ed-692f86bfccb7 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 247, 62, 20])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(2422046), primary: true, slot: 0 }], source_nat: SourceNatConfig { ip: 10.151.2.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.2.176), floating_ips: [], firewall_rules: [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 49ed10a6-8516-40ff-a573-5f419cb03dfc, kind: Instance { id: 1f8794d5-895b-43ae-93ed-692f86bfccb7 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 247, 62, 20])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(2422046), primary: true, slot: 0 }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 49ed10a6-8516-40ff-a573-5f419cb03dfc, kind: Instance { id: 1f8794d5-895b-43ae-93ed-692f86bfccb7 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 247, 62, 20])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(2422046), primary: true, slot: 0 }], filter_hosts: Some([Vpc(Vni(2422046))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 49ed10a6-8516-40ff-a573-5f419cb03dfc, kind: Instance { id: 1f8794d5-895b-43ae-93ed-692f86bfccb7 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 247, 62, 20])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(2422046), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-70d3fca55a4", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 17aa7c57-ea22-4897-a188-6ddd36f0db8a, read_only_parent: Some(Volume { block_size: 512, id: ecd5b714-8efd-47a5-a08c-a385d8595ac3, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 0e2134da-e682-4e76-a47d-4449a95a8edd, key: Some("YQ8Jzh6H6ZL7hVOL3Gy+lSQaGmePLhNYiDpTwThsZYw="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::14]:19001", "[fd00:1122:3344:101::19]:19001", "[fd00:1122:3344:101::17]:19002"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 17aa7c57-ea22-4897-a188-6ddd36f0db8a, key: Some("EJCYRHrF7MxWReI3qFkfDucLYfuuwo1P4Vin0J5Qj1E="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::17]:19003", "[fd00:1122:3344:101::19]:19002", "[fd00:1122:3344:101::12]:19000"] } }] } }], cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAAAWZU6uWK5YAABlTq5YAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAFmVOrliuWAAAZU6uWAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiIxZjg3OTRkNS04OTViLTQzYWUtOTNlZC02OTJmODZiZmNjYjciLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJRGdOUXZveW5vK0FPUldneDdEekdiR2NSTG1uSVZlT3VzZ1g3a1lhRkozRCJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA") } instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7 instance_runtime = InstanceRuntimeState { propolis_id: Some(f26a90e0-2719-418f-ba7b-06c7da6036f0), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-05-14T09:51:09.852064397Z } metadata = InstanceMetadata { silo_id: 82af3ca8-5091-4869-8146-a9112885b828, project_id: 4ed919d2-cc7b-4120-bf73-e8ea0ff719fe } propolis_addr = [fd00:1122:3344:101::1:0]:12400 propolis_id = f26a90e0-2719-418f-ba7b-06c7da6036f0 vmm_runtime = VmmRuntimeState { state: Starting, gen: Generation(1), time_updated: 2024-05-14T09:51:09.828338Z }
26472024-05-14T09:51:10.278ZINFOSledAgent (InstanceManager): registering new instance file = sled-agent/src/instance_manager.rs:580 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7
26482024-05-14T09:51:10.279ZINFOSledAgent (InstanceManager): initializing new Instance file = sled-agent/src/instance.rs:938 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7 propolis_id = f26a90e0-2719-418f-ba7b-06c7da6036f0 state = InstanceInitialState { hardware: InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: 49ed10a6-8516-40ff-a573-5f419cb03dfc, kind: Instance { id: 1f8794d5-895b-43ae-93ed-692f86bfccb7 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 247, 62, 20])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(2422046), primary: true, slot: 0 }], source_nat: SourceNatConfig { ip: 10.151.2.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.2.176), floating_ips: [], firewall_rules: [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 49ed10a6-8516-40ff-a573-5f419cb03dfc, kind: Instance { id: 1f8794d5-895b-43ae-93ed-692f86bfccb7 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 247, 62, 20])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(2422046), primary: true, slot: 0 }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 49ed10a6-8516-40ff-a573-5f419cb03dfc, kind: Instance { id: 1f8794d5-895b-43ae-93ed-692f86bfccb7 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 247, 62, 20])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(2422046), primary: true, slot: 0 }], filter_hosts: Some([Vpc(Vni(2422046))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 49ed10a6-8516-40ff-a573-5f419cb03dfc, kind: Instance { id: 1f8794d5-895b-43ae-93ed-692f86bfccb7 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 247, 62, 20])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(2422046), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-70d3fca55a4", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 17aa7c57-ea22-4897-a188-6ddd36f0db8a, read_only_parent: Some(Volume { block_size: 512, id: ecd5b714-8efd-47a5-a08c-a385d8595ac3, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 0e2134da-e682-4e76-a47d-4449a95a8edd, key: Some("YQ8Jzh6H6ZL7hVOL3Gy+lSQaGmePLhNYiDpTwThsZYw="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::14]:19001", "[fd00:1122:3344:101::19]:19001", "[fd00:1122:3344:101::17]:19002"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 17aa7c57-ea22-4897-a188-6ddd36f0db8a, key: Some("EJCYRHrF7MxWReI3qFkfDucLYfuuwo1P4Vin0J5Qj1E="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::17]:19003", "[fd00:1122:3344:101::19]:19002", "[fd00:1122:3344:101::12]:19000"] } }] } }], cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAAAWZU6uWK5YAABlTq5YAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAFmVOrliuWAAAZU6uWAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiIxZjg3OTRkNS04OTViLTQzYWUtOTNlZC02OTJmODZiZmNjYjciLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJRGdOUXZveW5vK0FPUldneDdEekdiR2NSTG1uSVZlT3VzZ1g3a1lhRkozRCJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA") }, instance_runtime: InstanceRuntimeState { propolis_id: Some(f26a90e0-2719-418f-ba7b-06c7da6036f0), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-05-14T09:51:09.852064397Z }, vmm_runtime: VmmRuntimeState { state: Starting, gen: Generation(1), time_updated: 2024-05-14T09:51:09.828338Z }, propolis_addr: [fd00:1122:3344:101::1:0]:12400 }
26492024-05-14T09:51:10.280ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 18244 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:32960 req_id = 91a65a15-a607-489f-8c04-98c11f6c1cd1 response_code = 200 uri = /instances/1f8794d5-895b-43ae-93ed-692f86bfccb7
26502024-05-14T09:51:10.389ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61674
26512024-05-14T09:51:10.391ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 2106 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:61674 req_id = c8416eb4-790a-4953-9a22-3d519dd0e5ce response_code = 200 uri = /instances/1f8794d5-895b-43ae-93ed-692f86bfccb7/state
26522024-05-14T09:51:10.415ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:394 port = Port { inner: PortInner { name: "opte6", ip: 172.30.0.5, mac: MacAddr6([168, 64, 37, 247, 62, 20]), slot: 0, vni: Vni { inner: 2422046 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.0.1 }, vnic: "vopte6" } }
26532024-05-14T09:51:10.415ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
26542024-05-14T09:51:10.452ZINFOSledAgent (InstanceManager): Configuring new Omicron zone: oxz_propolis-server_f26a90e0-2719-418f-ba7b-06c7da6036f0 file = illumos-utils/src/zone.rs:315 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7
26552024-05-14T09:51:10.474ZINFOSledAgent (InstanceManager): Installing Omicron zone: oxz_propolis-server_f26a90e0-2719-418f-ba7b-06c7da6036f0 file = illumos-utils/src/zone.rs:354 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7
26562024-05-14T09:51:11.853ZINFOSledAgent (InstanceManager): Profile for oxz_propolis-server_f26a90e0-2719-418f-ba7b-06c7da6036f0: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="system/illumos/propolis-server"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlInstance0"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::1:0"/> <propval type="astring" name="listen_port" value="12400"/> <propval type="astring" name="metric_addr" value="[fd00:1122:3344:101::b]:12221"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7
26572024-05-14T09:51:11.854ZINFOSledAgent (InstanceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7 zone = oxz_propolis-server_f26a90e0-2719-418f-ba7b-06c7da6036f0
26582024-05-14T09:51:12.052ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_f26a90e0-2719-418f-ba7b-06c7da6036f0") failed: Property not found. retry in 67.293982ms file = illumos-utils/src/svc.rs:36 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7 zone = oxz_propolis-server_f26a90e0-2719-418f-ba7b-06c7da6036f0
26592024-05-14T09:51:12.136ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_f26a90e0-2719-418f-ba7b-06c7da6036f0") failed: Property not found. retry in 53.723103ms file = illumos-utils/src/svc.rs:36 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7 zone = oxz_propolis-server_f26a90e0-2719-418f-ba7b-06c7da6036f0
26602024-05-14T09:51:12.205ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_f26a90e0-2719-418f-ba7b-06c7da6036f0") failed: Property not found. retry in 200.346745ms file = illumos-utils/src/svc.rs:36 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7 zone = oxz_propolis-server_f26a90e0-2719-418f-ba7b-06c7da6036f0
26612024-05-14T09:51:12.417ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_f26a90e0-2719-418f-ba7b-06c7da6036f0") failed: Property not found. retry in 249.482112ms file = illumos-utils/src/svc.rs:36 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7 zone = oxz_propolis-server_f26a90e0-2719-418f-ba7b-06c7da6036f0
26622024-05-14T09:51:12.682ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_f26a90e0-2719-418f-ba7b-06c7da6036f0") failed: Property not found. retry in 883.725022ms file = illumos-utils/src/svc.rs:36 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7 zone = oxz_propolis-server_f26a90e0-2719-418f-ba7b-06c7da6036f0
26632024-05-14T09:51:13.576ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_f26a90e0-2719-418f-ba7b-06c7da6036f0") failed: Property not found. retry in 1.156923119s file = illumos-utils/src/svc.rs:36 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7 zone = oxz_propolis-server_f26a90e0-2719-418f-ba7b-06c7da6036f0
26642024-05-14T09:51:14.743ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_f26a90e0-2719-418f-ba7b-06c7da6036f0") failed: Property not found. retry in 1.192502644s file = illumos-utils/src/svc.rs:36 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7 zone = oxz_propolis-server_f26a90e0-2719-418f-ba7b-06c7da6036f0
26652024-05-14T09:51:15.946ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_f26a90e0-2719-418f-ba7b-06c7da6036f0") failed: Property not found. retry in 793.881562ms file = illumos-utils/src/svc.rs:36 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7 zone = oxz_propolis-server_f26a90e0-2719-418f-ba7b-06c7da6036f0
26662024-05-14T09:51:16.752ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_f26a90e0-2719-418f-ba7b-06c7da6036f0") failed: Property not found. retry in 1.392568325s file = illumos-utils/src/svc.rs:36 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7 zone = oxz_propolis-server_f26a90e0-2719-418f-ba7b-06c7da6036f0
26672024-05-14T09:51:18.175ZINFOSledAgent (InstanceManager): Started propolis in zone: oxz_propolis-server_f26a90e0-2719-418f-ba7b-06c7da6036f0 file = sled-agent/src/instance.rs:1423 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7
26682024-05-14T09:51:18.184ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_f26a90e0-2719-418f-ba7b-06c7da6036f0") failed: Property not found. retry in 41.369963ms file = illumos-utils/src/svc.rs:36 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7
26692024-05-14T09:51:18.234ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_f26a90e0-2719-418f-ba7b-06c7da6036f0") failed: Property not found. retry in 128.483191ms file = illumos-utils/src/svc.rs:36 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7
26702024-05-14T09:51:18.373ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_f26a90e0-2719-418f-ba7b-06c7da6036f0") failed: Property not found. retry in 167.812163ms file = illumos-utils/src/svc.rs:36 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7
26712024-05-14T09:51:18.551ZINFOSledAgent (InstanceManager): Propolis SMF service is online file = sled-agent/src/instance.rs:1432 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7
26722024-05-14T09:51:18.593ZINFOSledAgent (InstanceManager): Propolis HTTP server online file = sled-agent/src/instance.rs:1446 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7
26732024-05-14T09:51:18.600ZINFOSledAgent (InstanceManager): Sending ensure request to propolis: InstanceEnsureRequest { cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAAAWZU6uWK5YAABlTq5YAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAFmVOrliuWAAAZU6uWAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiIxZjg3OTRkNS04OTViLTQzYWUtOTNlZC02OTJmODZiZmNjYjciLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJRGdOUXZveW5vK0FPUldneDdEekdiR2NSTG1uSVZlT3VzZ1g3a1lhRkozRCJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"), disks: [DiskRequest { device: "nvme", name: "disk-70d3fca55a4", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 17aa7c57-ea22-4897-a188-6ddd36f0db8a, read_only_parent: Some(Volume { block_size: 512, id: ecd5b714-8efd-47a5-a08c-a385d8595ac3, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 0e2134da-e682-4e76-a47d-4449a95a8edd, key: Some("YQ8Jzh6H6ZL7hVOL3Gy+lSQaGmePLhNYiDpTwThsZYw="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::14]:19001", "[fd00:1122:3344:101::19]:19001", "[fd00:1122:3344:101::17]:19002"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 17aa7c57-ea22-4897-a188-6ddd36f0db8a, key: Some("EJCYRHrF7MxWReI3qFkfDucLYfuuwo1P4Vin0J5Qj1E="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::17]:19003", "[fd00:1122:3344:101::19]:19002", "[fd00:1122:3344:101::12]:19000"] } }] } }], migrate: None, nics: [NetworkInterfaceRequest { name: "vopte6", slot: Slot(0) }], properties: InstanceProperties { bootrom_id: 00000000-0000-0000-0000-000000000000, description: "Test description", id: 1f8794d5-895b-43ae-93ed-692f86bfccb7, image_id: 00000000-0000-0000-0000-000000000000, memory: 1024, metadata: InstanceMetadata { project_id: 4ed919d2-cc7b-4120-bf73-e8ea0ff719fe, silo_id: 82af3ca8-5091-4869-8146-a9112885b828 }, name: "localshark", vcpus: 2 } } file = sled-agent/src/instance.rs:685 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7
26742024-05-14T09:51:18.785ZINFOSledAgent (InstanceManager): result of instance_ensure call is Ok(InstanceEnsureResponse { migrate: None }) file = sled-agent/src/instance.rs:687 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7
26752024-05-14T09:51:18.785ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Starting), migration_status: NoMigration, time: 2024-05-14T09:51:18.785934401Z } file = sled-agent/src/instance.rs:569 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7
26762024-05-14T09:51:18.792ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:591 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7 new_instance_state = InstanceRuntimeState { propolis_id: Some(f26a90e0-2719-418f-ba7b-06c7da6036f0), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-05-14T09:51:09.852064397Z } new_vmm_state = VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-05-14T09:51:18.785934401Z } propolis_id = f26a90e0-2719-418f-ba7b-06c7da6036f0
26772024-05-14T09:51:18.793ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:491 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(f26a90e0-2719-418f-ba7b-06c7da6036f0), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-05-14T09:51:09.852064397Z }, propolis_id: f26a90e0-2719-418f-ba7b-06c7da6036f0, vmm_state: VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-05-14T09:51:18.785934401Z } }
26782024-05-14T09:51:18.863ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Running), migration_status: NoMigration, time: 2024-05-14T09:51:18.863083630Z } file = sled-agent/src/instance.rs:569 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7
26792024-05-14T09:51:18.863ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:591 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7 new_instance_state = InstanceRuntimeState { propolis_id: Some(f26a90e0-2719-418f-ba7b-06c7da6036f0), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-05-14T09:51:09.852064397Z } new_vmm_state = VmmRuntimeState { state: Running, gen: Generation(3), time_updated: 2024-05-14T09:51:18.863083630Z } propolis_id = f26a90e0-2719-418f-ba7b-06c7da6036f0
26802024-05-14T09:51:18.863ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:491 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(f26a90e0-2719-418f-ba7b-06c7da6036f0), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-05-14T09:51:09.852064397Z }, propolis_id: f26a90e0-2719-418f-ba7b-06c7da6036f0, vmm_state: VmmRuntimeState { state: Running, gen: Generation(3), time_updated: 2024-05-14T09:51:18.863083630Z } }
26812024-05-14T09:51:20.943ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 837 local_addr = [fd00:1122:3344:101::1]:51153 method = GET remote_addr = [fd00:1122:3344:101::d]:45846 req_id = 1a85f77f-08c8-4ac7-abb7-76a9b8e6edd5 response_code = 200 sled_id = 5f617488-2ef8-4cf1-81f7-2b249f5b34c0 uri = //5f617488-2ef8-4cf1-81f7-2b249f5b34c0
26822024-05-14T09:51:27.698ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33779
26832024-05-14T09:51:27.700ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 1194 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:33779 req_id = 44402cf3-80de-403f-b31d-d70a40cc9376 response_code = 200 uri = /instances/1f8794d5-895b-43ae-93ed-692f86bfccb7/state
26842024-05-14T09:51:27.700ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Stopping), migration_status: NoMigration, time: 2024-05-14T09:51:27.700199451Z } file = sled-agent/src/instance.rs:569 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7
26852024-05-14T09:51:27.700ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:591 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7 new_instance_state = InstanceRuntimeState { propolis_id: Some(f26a90e0-2719-418f-ba7b-06c7da6036f0), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-05-14T09:51:09.852064397Z } new_vmm_state = VmmRuntimeState { state: Stopping, gen: Generation(5), time_updated: 2024-05-14T09:51:27.700199451Z } propolis_id = f26a90e0-2719-418f-ba7b-06c7da6036f0
26862024-05-14T09:51:27.700ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:491 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(f26a90e0-2719-418f-ba7b-06c7da6036f0), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-05-14T09:51:09.852064397Z }, propolis_id: f26a90e0-2719-418f-ba7b-06c7da6036f0, vmm_state: VmmRuntimeState { state: Stopping, gen: Generation(5), time_updated: 2024-05-14T09:51:27.700199451Z } }
26872024-05-14T09:51:27.851ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Destroyed), migration_status: NoMigration, time: 2024-05-14T09:51:27.851186076Z } file = sled-agent/src/instance.rs:569 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7
26882024-05-14T09:51:27.851ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:591 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7 new_instance_state = InstanceRuntimeState { propolis_id: None, dst_propolis_id: None, migration_id: None, gen: Generation(5), time_updated: 2024-05-14T09:51:27.851186076Z } new_vmm_state = VmmRuntimeState { state: Destroyed, gen: Generation(6), time_updated: 2024-05-14T09:51:27.851186076Z } propolis_id = f26a90e0-2719-418f-ba7b-06c7da6036f0
26892024-05-14T09:51:27.851ZINFOSledAgent (InstanceManager): terminating VMM that has exited file = sled-agent/src/instance.rs:606 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7
26902024-05-14T09:51:27.851ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
26912024-05-14T09:51:27.851ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
26922024-05-14T09:51:27.852ZINFOSledAgent (ZoneBundler): creating zone bundle context = ZoneBundleContext { storage_dirs: ["/pool/int/112d7827-193f-456c-94a5-19163d29475e/debug/bundle/zone", "/pool/int/e21e7560-2c98-46a4-bdda-f772e50ec2a3/debug/bundle/zone"], cause: TerminatedInstance, extra_log_dirs: ["/pool/ext/c919f4ac-100e-4c1e-bd24-76220f4848fc/crypt/debug", "/pool/ext/9d13b91d-5969-417a-bfea-ad89852618b9/crypt/debug", "/pool/ext/39005e7e-189e-49db-8463-a206621e7a54/crypt/debug", "/pool/ext/a7771067-edf5-47c9-a972-69320dc9ee62/crypt/debug", "/pool/ext/096e5012-bddf-482f-8ad5-65f3b0930676/crypt/debug", "/pool/ext/ae2f89f7-35c8-4c05-940c-cdee6d8f8a12/crypt/debug", "/pool/ext/42b2209d-f16a-4fca-a84d-54ecc9f13877/crypt/debug", "/pool/ext/95ce52af-b115-45d4-acd1-3598b0bafa5d/crypt/debug", "/pool/ext/ad348a98-27df-494e-af4b-560a8a421d64/crypt/debug"] } file = sled-agent/src/zone_bundle.rs:437 zone_name = oxz_propolis-server_f26a90e0-2719-418f-ba7b-06c7da6036f0
26932024-05-14T09:51:28.577ZINFOSledAgent (ZoneBundler): finished zone bundle file = sled-agent/src/zone_bundle.rs:1230 metadata = ZoneBundleMetadata { id: ZoneBundleId { zone_name: "oxz_propolis-server_f26a90e0-2719-418f-ba7b-06c7da6036f0", bundle_id: 60fb4ce8-86a3-4b8a-8e3d-f794976e1ff4 }, time_created: 2024-05-14T09:51:27.859795567Z, version: 0, cause: TerminatedInstance }
26942024-05-14T09:51:28.577ZWARNSledAgent (InstanceManager): Halting and removing zone: oxz_propolis-server_f26a90e0-2719-418f-ba7b-06c7da6036f0 file = sled-agent/src/instance.rs:780 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7
26952024-05-14T09:51:30.360ZINFOSledAgent (InstanceManager): halt_and_remove_logged: Previous zone state: Running file = illumos-utils/src/zone.rs:269 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7
26962024-05-14T09:51:30.388ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:491 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: None, dst_propolis_id: None, migration_id: None, gen: Generation(5), time_updated: 2024-05-14T09:51:27.851186076Z }, propolis_id: f26a90e0-2719-418f-ba7b-06c7da6036f0, vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-05-14T09:51:30.388417655Z } }
26972024-05-14T09:51:30.486ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:34723
26982024-05-14T09:51:30.491ZWARNSledAgent (PortManager): unset_virtual_nic_host unimplmented file = illumos-utils/src/opte/port_manager.rs:620
26992024-05-14T09:51:30.491ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 5520 local_addr = [fd00:1122:3344:101::1]:12345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:34723 req_id = 1cb222ce-26e1-403b-80f5-6572a8eb856b response_code = 204 uri = /v2p/49ed10a6-8516-40ff-a573-5f419cb03dfc
27002024-05-14T09:51:30.894ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:491 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: None, dst_propolis_id: None, migration_id: None, gen: Generation(5), time_updated: 2024-05-14T09:51:27.851186076Z }, propolis_id: f26a90e0-2719-418f-ba7b-06c7da6036f0, vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-05-14T09:51:30.388417655Z } }
27012024-05-14T09:51:30.898ZINFOSledAgent (InstanceManager): State monitoring task complete file = sled-agent/src/instance.rs:723 instance_id = 1f8794d5-895b-43ae-93ed-692f86bfccb7
27022024-05-14T09:51:31.372ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:41328
27032024-05-14T09:51:31.372ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 43 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:41328 req_id = 2d985902-28d6-46b6-a7a2-349824e4e39a response_code = 200 uri = /baseboard
27042024-05-14T09:51:33.609ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:47283
27052024-05-14T09:51:33.659ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 49602 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:47283 req_id = c8898bd8-94ea-45a3-ae77-dd31b7bbf690 response_code = 204 uri = /switch-ports
27062024-05-14T09:51:33.659ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:45736
27072024-05-14T09:51:33.705ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 45755 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:45736 req_id = a0c3f921-db6c-46ed-adea-9b002a3ffeee response_code = 204 uri = /switch-ports
27082024-05-14T09:51:33.705ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 98 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:47283 req_id = 0b212418-d101-462c-93cf-4c0dcba53169 response_code = 200 uri = /network-bootstore-config
27092024-05-14T09:51:33.705ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 132 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:45736 req_id = 1bfe3946-885d-4d28-8c50-b30d53582642 response_code = 200 uri = /network-bootstore-config
27102024-05-14T09:51:33.728ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:60425
27112024-05-14T09:51:33.773ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 45280 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:60425 req_id = e2df5cb8-5775-471b-bce0-8d23b65e75ae response_code = 204 uri = /switch-ports
27122024-05-14T09:51:33.773ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/283d897/dropshot/src/server.rs:878 latency_us = 146 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:60425 req_id = d92334be-99be-429c-a1ff-6befd07fc546 response_code = 200 uri = /network-bootstore-config