Max level shown:
1 [ Mar 29 05:52:48 Enabled. ]
2 [ Mar 29 05:52:48 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]
3 [ Mar 29 05:52:48 Method "start" exited with status 0. ]
4 [ Mar 29 05:52:48 Rereading configuration. ]
5 [ Mar 29 05:52:48 No 'refresh' method defined. Treating as :true. ]
6 note: configured to log to "/dev/stdout"
72024-03-29T05:52:48.856ZINFOSledAgent: Ensuring contract reaper thread file = illumos-utils/src/running_zone.rs:163
82024-03-29T05:52:48.856ZINFOSledAgent (BootstrapAgentStartup): finding links ["net0", "net1"] file = sled-agent/src/bootstrap/pre_server.rs:253
92024-03-29T05:52:48.910ZINFOSledAgent (BootstrapAgentStartup): Starting mg-ddm service addr-objs = [AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }] file = sled-agent/src/bootstrap/pre_server.rs:260
102024-03-29T05:52:48.910ZINFOSledAgent (BootstrapAgentStartup): Importing mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:42 path = /opt/oxide/mg-ddm/pkg/ddm/manifest.xml
112024-03-29T05:52:49.028ZINFOSledAgent (BootstrapAgentStartup): Setting mg-ddm interfaces file = sled-agent/src/bootstrap/maghemite.rs:50 interfaces = ("net0/ll" "net1/ll")
122024-03-29T05:52:49.037ZINFOSledAgent (BootstrapAgentStartup): Enabling mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:56
132024-03-29T05:52:49.045ZINFOSledAgent (BootstrapAgentStartup): Ensuring zfs key directory exists file = sled-agent/src/bootstrap/pre_server.rs:270 path = /var/run/oxide/
142024-03-29T05:52:49.651ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:87 prefix = Ipv6Prefix { addr: fdb0:200::, len: 64 }
152024-03-29T05:52:49.793ZINFOSledAgent: Starting KeyManager file = sled-agent/src/long_running_tasks.rs:115
162024-03-29T05:52:49.793ZINFOSledAgent: Starting StorageManager file = sled-agent/src/long_running_tasks.rs:127
172024-03-29T05:52:49.793ZINFOSledAgent: Starting StorageMonitor file = sled-agent/src/long_running_tasks.rs:137
182024-03-29T05:52:49.793ZINFOSledAgent: Starting HardwareManager file = sled-agent/src/long_running_tasks.rs:157 sled_mode = Scrimlet { asic: SoftNpuZone }
192024-03-29T05:52:49.793ZINFOSledAgent (DumpSetup-archival): DumpSetup poll loop started. file = sled-agent/src/dump_setup.rs:291
202024-03-29T05:52:49.793ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:801
212024-03-29T05:52:49.793ZINFOSledAgent (HardwareManager): Creating HardwareManager file = sled-hardware/src/illumos/mod.rs:609
222024-03-29T05:52:49.796ZINFOSledAgent (HardwareManager): Generated i86pc baseboard Pc { identifier: "buskin", model: "i86pc" } file = sled-hardware/src/illumos/mod.rs:535
232024-03-29T05:52:49.796ZWARNSledAgent (HardwareManager): Device is not a Gimlet (i86pc), proceeding with null hardware view file = sled-hardware/src/illumos/mod.rs:658
242024-03-29T05:52:49.796ZINFOSledAgent: Starting HardwareMonitor file = sled-agent/src/long_running_tasks.rs:171
252024-03-29T05:52:49.796ZINFOSledAgent (HardwareManager): Monitoring for hardware updates file = sled-hardware/src/illumos/mod.rs:716
262024-03-29T05:52:49.796ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:227 vdev = /scratch/m2_0.vdev
272024-03-29T05:52:49.796ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot file = sled-agent/src/hardware_monitor.rs:261 underlay_network_info = None
282024-03-29T05:52:49.796ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }), tx: ... } file = sled-storage/src/manager.rs:381
292024-03-29T05:52:49.796ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }
302024-03-29T05:52:49.797ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:111 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" } path = /scratch/m2_0.vdev
312024-03-29T05:52:49.809ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_0.vdev file = sled-hardware/src/disk.rs:301
322024-03-29T05:52:49.809ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:312 id = 79ce42ff-9813-469e-9253-44dff191e19b
332024-03-29T05:52:49.911ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 79ce42ff-9813-469e-9253-44dff191e19b, kind: Internal }
342024-03-29T05:52:50.123ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:381
352024-03-29T05:52:50.123ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:227 vdev = /scratch/m2_1.vdev
362024-03-29T05:52:50.123ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }
372024-03-29T05:52:50.123ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }), tx: ... } file = sled-storage/src/manager.rs:381
382024-03-29T05:52:50.123ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }
392024-03-29T05:52:50.123ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:111 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" } path = /scratch/m2_1.vdev
402024-03-29T05:52:50.132ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_1.vdev file = sled-hardware/src/disk.rs:301
412024-03-29T05:52:50.132ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:312 id = 615b4df3-8040-4d04-947b-807c077fd24b
422024-03-29T05:52:50.221ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 615b4df3-8040-4d04-947b-807c077fd24b, kind: Internal }
432024-03-29T05:52:50.428ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:227 vdev = /scratch/u2_0.vdev
442024-03-29T05:52:50.428ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 79ce42ff-9813-469e-9253-44dff191e19b, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 615b4df3-8040-4d04-947b-807c077fd24b, kind: Internal } }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
452024-03-29T05:52:50.428ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 }), tx: ... } file = sled-storage/src/manager.rs:381
462024-03-29T05:52:50.428ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }
472024-03-29T05:52:50.428ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:227 vdev = /scratch/u2_1.vdev
482024-03-29T05:52:50.428ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 79ce42ff-9813-469e-9253-44dff191e19b, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 615b4df3-8040-4d04-947b-807c077fd24b, kind: Internal } }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
492024-03-29T05:52:50.428ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 }), tx: ... } file = sled-storage/src/manager.rs:381
502024-03-29T05:52:50.428ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }
512024-03-29T05:52:50.428ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:227 vdev = /scratch/u2_2.vdev
522024-03-29T05:52:50.428ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 }), tx: ... } file = sled-storage/src/manager.rs:381
532024-03-29T05:52:50.428ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 79ce42ff-9813-469e-9253-44dff191e19b, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 615b4df3-8040-4d04-947b-807c077fd24b, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
542024-03-29T05:52:50.428ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }
552024-03-29T05:52:50.428ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:227 vdev = /scratch/u2_3.vdev
562024-03-29T05:52:50.428ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 79ce42ff-9813-469e-9253-44dff191e19b, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 615b4df3-8040-4d04-947b-807c077fd24b, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
572024-03-29T05:52:50.428ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 }), tx: ... } file = sled-storage/src/manager.rs:381
582024-03-29T05:52:50.428ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }
592024-03-29T05:52:50.428ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:227 vdev = /scratch/u2_4.vdev
602024-03-29T05:52:50.428ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, variant: U2, slot: 1030 }), tx: ... } file = sled-storage/src/manager.rs:381
612024-03-29T05:52:50.428ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }
622024-03-29T05:52:50.428ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 79ce42ff-9813-469e-9253-44dff191e19b, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 615b4df3-8040-4d04-947b-807c077fd24b, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
632024-03-29T05:52:50.428ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:227 vdev = /scratch/u2_5.vdev
642024-03-29T05:52:50.428ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, variant: U2, slot: 1031 }), tx: ... } file = sled-storage/src/manager.rs:381
652024-03-29T05:52:50.428ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }
662024-03-29T05:52:50.428ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 79ce42ff-9813-469e-9253-44dff191e19b, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 615b4df3-8040-4d04-947b-807c077fd24b, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, variant: U2, slot: 1030 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
672024-03-29T05:52:50.428ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:227 vdev = /scratch/u2_6.vdev
682024-03-29T05:52:50.428ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }, variant: U2, slot: 1032 }), tx: ... } file = sled-storage/src/manager.rs:381
692024-03-29T05:52:50.428ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }
702024-03-29T05:52:50.428ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 79ce42ff-9813-469e-9253-44dff191e19b, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 615b4df3-8040-4d04-947b-807c077fd24b, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, variant: U2, slot: 1030 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, variant: U2, slot: 1031 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
712024-03-29T05:52:50.428ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:227 vdev = /scratch/u2_7.vdev
722024-03-29T05:52:50.428ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }, variant: U2, slot: 1033 }), tx: ... } file = sled-storage/src/manager.rs:381
732024-03-29T05:52:50.428ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }
742024-03-29T05:52:50.428ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 79ce42ff-9813-469e-9253-44dff191e19b, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 615b4df3-8040-4d04-947b-807c077fd24b, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, variant: U2, slot: 1030 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, variant: U2, slot: 1031 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }, variant: U2, slot: 1032 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
752024-03-29T05:52:50.428ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:227 vdev = /scratch/u2_8.vdev
762024-03-29T05:52:50.428ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }, variant: U2, slot: 1034 }), tx: ... } file = sled-storage/src/manager.rs:381
772024-03-29T05:52:50.428ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }
782024-03-29T05:52:50.428ZINFOSledAgent: Waiting for boot disk file = sled-agent/src/long_running_tasks.rs:87
792024-03-29T05:52:50.428ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 79ce42ff-9813-469e-9253-44dff191e19b, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 615b4df3-8040-4d04-947b-807c077fd24b, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, variant: U2, slot: 1030 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, variant: U2, slot: 1031 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }, variant: U2, slot: 1032 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }, variant: U2, slot: 1033 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
802024-03-29T05:52:50.428ZINFOSledAgent: Found boot disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" } file = sled-agent/src/long_running_tasks.rs:89
812024-03-29T05:52:50.428ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
822024-03-29T05:52:50.428ZINFOSledAgent: Starting Bootstore file = sled-agent/src/long_running_tasks.rs:195
832024-03-29T05:52:50.428ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 79ce42ff-9813-469e-9253-44dff191e19b, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 615b4df3-8040-4d04-947b-807c077fd24b, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, variant: U2, slot: 1030 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, variant: U2, slot: 1031 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }, variant: U2, slot: 1032 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }, variant: U2, slot: 1033 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }, variant: U2, slot: 1034 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
842024-03-29T05:52:50.429ZINFOSledAgent (bootstore): No ledger in /pool/int/79ce42ff-9813-469e-9253-44dff191e19b/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
852024-03-29T05:52:50.429ZINFOSledAgent (bootstore): No ledger in /pool/int/615b4df3-8040-4d04-947b-807c077fd24b/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
862024-03-29T05:52:50.429ZINFOSledAgent (bootstore): No ledger found. Loading Fsm::State as Uninitialized file = bootstore/src/schemes/v0/storage.rs:83 peer_id = pc-buskin-i86pc
872024-03-29T05:52:50.429ZINFOSledAgent (bootstore): No ledger in /pool/int/79ce42ff-9813-469e-9253-44dff191e19b/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
882024-03-29T05:52:50.429ZINFOSledAgent (bootstore): No ledger in /pool/int/615b4df3-8040-4d04-947b-807c077fd24b/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
892024-03-29T05:52:50.429ZINFOSledAgent (bootstore): No ledger found for network config file = bootstore/src/schemes/v0/storage.rs:146 peer_id = pc-buskin-i86pc
902024-03-29T05:52:50.429ZINFOSledAgent: Starting Bootstore DDMD poller file = sled-agent/src/long_running_tasks.rs:200
912024-03-29T05:52:50.429ZINFOSledAgent: Starting ZoneBundler related tasks file = sled-agent/src/long_running_tasks.rs:215
922024-03-29T05:52:50.439ZINFOSledAgent (ServiceManager): Creating ServiceManager file = sled-agent/src/services.rs:701
932024-03-29T05:52:50.439ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:289 next_cleanup = Instant { tv_sec: 1304, tv_nsec: 332475260 } time_to_next_cleanup = 599.999918738s
942024-03-29T05:52:50.439ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3588
952024-03-29T05:52:50.439ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
962024-03-29T05:52:50.439ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/79ce42ff-9813-469e-9253-44dff191e19b/config/sled-agent-request.json file = common/src/ledger.rs:188
972024-03-29T05:52:50.439ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/615b4df3-8040-4d04-947b-807c077fd24b/config/sled-agent-request.json file = common/src/ledger.rs:188
982024-03-29T05:52:50.442ZINFOSledAgent (dropshot (BootstrapAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:195 local_addr = [fdb0:200::1]:80
992024-03-29T05:52:50.443ZINFOSledAgent (SledAgentSprocketsServer): Started listening file = sled-agent/src/bootstrap/sprockets_server.rs:44 local_addr = [fdb0:200::1]:12346
1002024-03-29T05:52:50.443ZINFOSledAgent (RSS): Injecting RSS configuration: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.1.164, last: 10.151.1.173, }, ), ], external_dns_ips: [ 10.151.1.164, 10.151.1.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV1 { rack_subnet: Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56, }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV1 { routes: [ RouteConfig { destination: V4( Ipv4Network { addr: 0.0.0.0, prefix: 0, }, ), nexthop: 10.151.1.100, }, ], addresses: [ V4( Ipv4Network { addr: 10.151.1.174, prefix: 24, }, ), ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, }, ], bgp: [], bfd: [], }, } file = sled-agent/src/rack_setup/service.rs:997
1012024-03-29T05:52:50.443ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
1022024-03-29T05:52:50.443ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1032024-03-29T05:52:50.443ZINFOSledAgent (RSS): No ledger in /pool/int/79ce42ff-9813-469e-9253-44dff191e19b/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1042024-03-29T05:52:50.443ZINFOSledAgent (RSS): No ledger in /pool/int/615b4df3-8040-4d04-947b-807c077fd24b/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1052024-03-29T05:52:50.443ZINFOSledAgent (RSS): RSS configuration has not been fully applied yet file = sled-agent/src/rack_setup/service.rs:1057
1062024-03-29T05:52:50.443ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1072024-03-29T05:52:50.443ZINFOSledAgent (RSS): No ledger in /pool/int/79ce42ff-9813-469e-9253-44dff191e19b/config/rss-sled-plan.json file = common/src/ledger.rs:188
1082024-03-29T05:52:50.443ZINFOSledAgent (RSS): No ledger in /pool/int/615b4df3-8040-4d04-947b-807c077fd24b/config/rss-sled-plan.json file = common/src/ledger.rs:188
1092024-03-29T05:52:50.443ZINFOSledAgent (RSS): Creating new allocation plan file = sled-agent/src/rack_setup/service.rs:1098
1102024-03-29T05:52:50.443ZINFOSledAgent (RSS): Serializing plan file = sled-agent/src/rack_setup/plan/sled.rs:118
1112024-03-29T05:52:50.443ZINFOSledAgent (RSS): Creating plan for the sled at fdb0:200::1 file = sled-agent/src/rack_setup/plan/sled.rs:91
1122024-03-29T05:52:50.443ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1132024-03-29T05:52:50.443ZINFOSledAgent (RSS): Writing ledger to /pool/int/79ce42ff-9813-469e-9253-44dff191e19b/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1142024-03-29T05:52:50.444ZINFOSledAgent (RSS): Writing ledger to /pool/int/615b4df3-8040-4d04-947b-807c077fd24b/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1152024-03-29T05:52:50.444ZINFOSledAgent (RSS): Sled plan written to storage file = sled-agent/src/rack_setup/plan/sled.rs:138
1162024-03-29T05:52:50.444ZINFOSledAgent (RSS): Writing Rack Network Configuration to bootstore file = sled-agent/src/rack_setup/service.rs:1131
1172024-03-29T05:52:50.444ZINFOSledAgent (bootstore): Attempting to update network config with generation: 1, current_generation: 0 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-buskin-i86pc
1182024-03-29T05:52:50.444ZINFOSledAgent (bootstore): Writing ledger to /pool/int/79ce42ff-9813-469e-9253-44dff191e19b/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
1192024-03-29T05:52:50.444ZINFOSledAgent (bootstore): Writing ledger to /pool/int/615b4df3-8040-4d04-947b-807c077fd24b/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
1202024-03-29T05:52:50.444ZINFOSledAgent (bootstore): Broadcasting network config with generation 1 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-buskin-i86pc
1212024-03-29T05:52:50.444ZINFOSledAgent (BootstrapAgentRssHandler): Received initialization request from RSS file = sled-agent/src/bootstrap/rss_handle.rs:217 request = StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 637eb488-60d7-4bc3-86a0-d192a08d81a4, rack_id: 8e0e1f9d-082c-48ca-86bf-5263e49ec430, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net(Ipv6Network { addr: fd00:1122:3344:101::, prefix: 64 }) } } } target_sled = [fdb0:200::1]:12346
1222024-03-29T05:52:50.444ZINFOSledAgent (SledAgentSprocketsServer): Accepted connection file = sled-agent/src/bootstrap/sprockets_server.rs:66 remote_addr = [fdb0:200::1]:43927
1232024-03-29T05:52:50.444ZINFOSledAgent (SledAgentMain): Loading Sled Agent: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 637eb488-60d7-4bc3-86a0-d192a08d81a4, rack_id: 8e0e1f9d-082c-48ca-86bf-5263e49ec430, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net(Ipv6Network { addr: fd00:1122:3344:101::, prefix: 64 }) } } } file = sled-agent/src/bootstrap/server.rs:356
1242024-03-29T05:52:50.444ZINFOSledAgent (SledAgentMain): KeyManager: using hardcoded secret retriever file = sled-agent/src/bootstrap/server.rs:371
1252024-03-29T05:52:50.444ZINFOSledAgent (StorageManager): Received KeyManagerReady file = sled-storage/src/manager.rs:381
1262024-03-29T05:52:50.444ZINFOSledAgent: setting up sled agent server file = sled-agent/src/server.rs:44
1272024-03-29T05:52:50.444ZINFOSledAgent (StorageManager): Loading storage config file = sled-storage/src/manager.rs:530
1282024-03-29T05:52:50.444ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
1292024-03-29T05:52:50.444ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:87 prefix = Ipv6Prefix { addr: fd00:1122:3344:101::, len: 64 }
1302024-03-29T05:52:50.444ZINFOSledAgent (StorageManager): No ledger in /pool/int/79ce42ff-9813-469e-9253-44dff191e19b/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_ledger
1312024-03-29T05:52:50.444ZINFOSledAgent (StorageManager): No ledger in /pool/int/615b4df3-8040-4d04-947b-807c077fd24b/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_ledger
1322024-03-29T05:52:50.444ZINFOSledAgent (StorageManager): No ledger of physical disks exists file = sled-storage/src/manager.rs:519
1332024-03-29T05:52:50.444ZINFOSledAgent (StorageManager): KeyManager ready, but no ledger detected file = sled-storage/src/manager.rs:547
1342024-03-29T05:52:50.447ZINFOSledAgent (ServiceManager): Enabling switch zone (new) file = sled-agent/src/services.rs:3819
1352024-03-29T05:52:50.457ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }
1362024-03-29T05:52:50.468ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }
1372024-03-29T05:52:50.479ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }
1382024-03-29T05:52:50.488ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }
1392024-03-29T05:52:50.498ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }
1402024-03-29T05:52:50.506ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }
1412024-03-29T05:52:50.515ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }
1422024-03-29T05:52:50.525ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }
1432024-03-29T05:52:50.534ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }
1442024-03-29T05:52:50.534ZINFOSledAgent (StorageManager): No disks to be automatically managed file = sled-storage/src/manager.rs:558
1452024-03-29T05:52:50.534ZINFOSledAgent (StorageManager): automatically managing disks file = sled-storage/src/manager.rs:370
1462024-03-29T05:52:50.534ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:339
1472024-03-29T05:52:50.534ZINFOSledAgent (StorageManager): Successfully synchronized disks without error file = sled-storage/src/manager.rs:449 result = DisksManagementResult { status: [] }
1482024-03-29T05:52:50.534ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1492024-03-29T05:52:50.555ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:315
1502024-03-29T05:52:50.572ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:354
1512024-03-29T05:52:50.585ZINFOSledAgent: SledAgent::new(..) starting file = sled-agent/src/sled_agent.rs:355 sled_id = 637eb488-60d7-4bc3-86a0-d192a08d81a4
1522024-03-29T05:52:50.585ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1532024-03-29T05:52:50.585ZINFOSledAgent: Requested swap device of size 64 GiB file = sled-agent/src/sled_agent.rs:367 sled_id = 637eb488-60d7-4bc3-86a0-d192a08d81a4
1542024-03-29T05:52:50.585ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1552024-03-29T05:52:50.588ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 66.217954ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "0ce11e9c-280a-4a78-b5ca-e9bf0d809d08", "content-length": "124", "date": "Fri, 29 Mar 2024 05:52:49 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "0ce11e9c-280a-4a78-b5ca-e9bf0d809d08" } file = clients/ddm-admin-client/src/lib.rs:178
1562024-03-29T05:52:50.599ZINFOSledAgent: attempting to create encrypted zvol: name="oxi_79ce42ff-9813-469e-9253-44dff191e19b/swap", size_gb=64 file = sled-agent/src/swap_device.rs:153
1572024-03-29T05:52:50.624ZINFOSledAgent: successfully created encrypted zvol: name="oxi_79ce42ff-9813-469e-9253-44dff191e19b/swap", size_gb=64 file = sled-agent/src/swap_device.rs:249
1582024-03-29T05:52:50.624ZINFOSledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_79ce42ff-9813-469e-9253-44dff191e19b/swap" file = sled-agent/src/swap_device.rs:96
1592024-03-29T05:52:50.629ZINFOSledAgent: Mounting backing filesystems file = sled-agent/src/sled_agent.rs:382 sled_id = 637eb488-60d7-4bc3-86a0-d192a08d81a4
1602024-03-29T05:52:50.629ZINFOSledAgent (BackingFs): Processing fmd file = sled-agent/src/backing_fs.rs:126
1612024-03-29T05:52:50.629ZINFOSledAgent (BackingFs): Ensuring dataset oxi_79ce42ff-9813-469e-9253-44dff191e19b/backing/fmd file = sled-agent/src/backing_fs.rs:136
1622024-03-29T05:52:50.656ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1632024-03-29T05:52:50.657ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 72.612163ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "84a878d9-bbc8-41ce-a540-f05687ccdbef", "content-length": "124", "date": "Fri, 29 Mar 2024 05:52:50 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "84a878d9-bbc8-41ce-a540-f05687ccdbef" } file = clients/ddm-admin-client/src/lib.rs:178
1642024-03-29T05:52:50.700ZINFOSledAgent (BackingFs): Stopping service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:170
1652024-03-29T05:52:50.731ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1662024-03-29T05:52:50.731ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 84.700671ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "0997fa07-2a66-496e-8349-7a2ef5d3c540", "content-length": "124", "date": "Fri, 29 Mar 2024 05:52:50 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "0997fa07-2a66-496e-8349-7a2ef5d3c540" } file = clients/ddm-admin-client/src/lib.rs:178
1672024-03-29T05:52:50.817ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1682024-03-29T05:52:50.818ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 191.407148ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "a9ece29e-ba4b-478e-8d0c-622e96a22f11", "content-length": "124", "date": "Fri, 29 Mar 2024 05:52:50 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "a9ece29e-ba4b-478e-8d0c-622e96a22f11" } file = clients/ddm-admin-client/src/lib.rs:178
1692024-03-29T05:52:50.919ZINFOSledAgent (BackingFs): Mounting oxi_79ce42ff-9813-469e-9253-44dff191e19b/backing/fmd on /var/fm/fmd file = sled-agent/src/backing_fs.rs:178
1702024-03-29T05:52:50.931ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/rsrc file = sled-agent/src/backing_fs.rs:186
1712024-03-29T05:52:50.931ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/ckpt file = sled-agent/src/backing_fs.rs:186
1722024-03-29T05:52:50.931ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/xprt file = sled-agent/src/backing_fs.rs:186
1732024-03-29T05:52:50.931ZINFOSledAgent (BackingFs): Starting service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:192
1742024-03-29T05:52:51.010ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1752024-03-29T05:52:51.011ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 203.161182ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "f8cdef98-de79-45cf-818b-e1ca2d4a6fb3", "content-length": "124", "date": "Fri, 29 Mar 2024 05:52:50 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "f8cdef98-de79-45cf-818b-e1ca2d4a6fb3" } file = clients/ddm-admin-client/src/lib.rs:178
1762024-03-29T05:52:51.215ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1772024-03-29T05:52:51.215ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 367.418485ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "74cb9cdc-8558-4db9-85cb-98647501d901", "content-length": "124", "date": "Fri, 29 Mar 2024 05:52:50 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "74cb9cdc-8558-4db9-85cb-98647501d901" } file = clients/ddm-admin-client/src/lib.rs:178
1782024-03-29T05:52:51.584ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1792024-03-29T05:52:51.585ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 362.961788ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "63786f76-aaf0-4036-aaa2-7e4a6d63fb3c", "content-length": "124", "date": "Fri, 29 Mar 2024 05:52:50 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "63786f76-aaf0-4036-aaa2-7e4a6d63fb3c" } file = clients/ddm-admin-client/src/lib.rs:178
1802024-03-29T05:52:51.635ZINFOSledAgent: using '[AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }]' as data links for xde driver file = illumos-utils/src/opte/illumos.rs:90 sled_id = 637eb488-60d7-4bc3-86a0-d192a08d81a4
1812024-03-29T05:52:51.680ZINFOSledAgent (VmmReservoirManager): 50% of 102952026112 physical ram = 51476013056 bytes) file = sled-agent/src/vmm_reservoir.rs:266 sled_id = 637eb488-60d7-4bc3-86a0-d192a08d81a4
1822024-03-29T05:52:51.680ZINFOSledAgent (VmmReservoirManager): Setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:274 sled_id = 637eb488-60d7-4bc3-86a0-d192a08d81a4
1832024-03-29T05:52:51.680ZINFOSledAgent (ServiceManager): sled agent started file = sled-agent/src/services.rs:983 underlay_address = fd00:1122:3344:101::1
1842024-03-29T05:52:51.680ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
1852024-03-29T05:52:51.681ZINFOSledAgent (dropshot (SledAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:195 local_addr = [fd00:1122:3344:101::1]:12345
1862024-03-29T05:52:51.681ZINFOSledAgent (SledAgentMain): Sled Agent loaded; recording configuration file = sled-agent/src/bootstrap/server.rs:426
1872024-03-29T05:52:51.681ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1882024-03-29T05:52:51.681ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/79ce42ff-9813-469e-9253-44dff191e19b/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
1892024-03-29T05:52:51.681ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/615b4df3-8040-4d04-947b-807c077fd24b/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
1902024-03-29T05:52:51.681ZWARNSledAgent: failed to register as a metric producer with Nexus error = "Metric registration error: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/metrics/producers): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/sled_agent.rs:1178 retry_after = 353.102767ms sled_id = 637eb488-60d7-4bc3-86a0-d192a08d81a4
1912024-03-29T05:52:51.681ZINFOSledAgent (HardwareMonitor): Sled Agent Started file = sled-agent/src/hardware_monitor.rs:138
1922024-03-29T05:52:51.681ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot file = sled-agent/src/hardware_monitor.rs:261 underlay_network_info = Some((fd00:1122:3344:101::2, Some(RackNetworkConfigV1 { rack_subnet: Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56 }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [PortConfigV1 { routes: [RouteConfig { destination: V4(Ipv4Network { addr: 0.0.0.0, prefix: 0 }), nexthop: 10.151.1.100 }], addresses: [V4(Ipv4Network { addr: 10.151.1.174, prefix: 24 })], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false }], bgp: [], bfd: [] })))
1932024-03-29T05:52:51.681ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3588
1942024-03-29T05:52:51.681ZINFOSledAgent (SledAgentSprocketsServer): Connection closed file = sled-agent/src/bootstrap/sprockets_server.rs:73 remote_addr = [fdb0:200::1]:43927
1952024-03-29T05:52:51.681ZINFOSledAgent (BootstrapAgentRssHandler): Peer agent initialized file = sled-agent/src/bootstrap/rss_handle.rs:108 peer = [fdb0:200::1]:12346
1962024-03-29T05:52:51.681ZINFOSledAgent (BootstrapAgentRssHandler): Initialized sled agent file = sled-agent/src/bootstrap/rss_handle.rs:232 target_sled = [fdb0:200::1]:12346
1972024-03-29T05:52:51.860ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1982024-03-29T05:52:51.860ZINFOSledAgent (RSS): No ledger in /pool/int/79ce42ff-9813-469e-9253-44dff191e19b/config/rss-service-plan-v3.json file = common/src/ledger.rs:188
1992024-03-29T05:52:51.860ZINFOSledAgent (RSS): No ledger in /pool/int/615b4df3-8040-4d04-947b-807c077fd24b/config/rss-service-plan-v3.json file = common/src/ledger.rs:188
2002024-03-29T05:52:51.860ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2012024-03-29T05:52:51.860ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2022024-03-29T05:52:51.865ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/637eb488-60d7-4bc3-86a0-d192a08d81a4): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 637eb488-60d7-4bc3-86a0-d192a08d81a4
2032024-03-29T05:52:51.892ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:38780
2042024-03-29T05:52:51.892ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2052024-03-29T05:52:51.893ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 264 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:38780 req_id = 97b2e42b-5240-4c32-9823-af8bb76c82b4 response_code = 200 uri = /inventory
2062024-03-29T05:52:51.924ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:49664
2072024-03-29T05:52:51.924ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 45 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:49664 req_id = 45e3715a-6597-4595-ab27-abf572680d25 response_code = 200 uri = /sled-role
2082024-03-29T05:52:51.925ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2092024-03-29T05:52:51.925ZINFOSledAgent (RSS): Writing ledger to /pool/int/79ce42ff-9813-469e-9253-44dff191e19b/config/.rss-service-plan-v3.json.tmp file = common/src/ledger.rs:199
2102024-03-29T05:52:51.925ZINFOSledAgent (RSS): Writing ledger to /pool/int/615b4df3-8040-4d04-947b-807c077fd24b/config/.rss-service-plan-v3.json.tmp file = common/src/ledger.rs:199
2112024-03-29T05:52:51.926ZINFOSledAgent (RSS): Service plan written to storage file = sled-agent/src/rack_setup/plan/service.rs:802
2122024-03-29T05:52:51.949ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
2132024-03-29T05:52:51.957ZINFOSledAgent (RSS): attempting to set up sled's storage: OmicronPhysicalDisksConfig { disks: [OmicronPhysicalDiskConfig { id: e108eec0-8a2a-4701-9cb4-4558ad340232, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, pool_id: ce966ff9-88a5-4653-bfe9-0d3a7a2ec9cc }, OmicronPhysicalDiskConfig { id: 51e3dbcf-e17a-4662-a949-da2a639e7b5e, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, pool_id: 97f2d604-fcf7-497f-8f26-7e660b2d684b }, OmicronPhysicalDiskConfig { id: 00216544-e869-44b2-95ee-cba360da3413, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, pool_id: 59af6774-3ad5-48ca-b81f-bfe9a37a0d07 }, OmicronPhysicalDiskConfig { id: 5794b57f-31b0-4298-bbda-d8bf0042564d, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, pool_id: 78643ff1-d165-41d4-8bbd-429ec8539197 }, OmicronPhysicalDiskConfig { id: a7713361-ccff-4308-af3b-3951fb7b4a1b, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, pool_id: cf82edba-c6a9-4917-87f1-2b21f22c3500 }, OmicronPhysicalDiskConfig { id: ab4e69e3-de5d-45fc-8b01-5f8bc0833418, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, pool_id: eaa368be-d4d5-4952-8370-0d8467190389 }, OmicronPhysicalDiskConfig { id: d9ebd94c-1f2e-4b96-b7f5-2b50ac7ecb6b, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }, pool_id: 2cb4ef32-43de-4203-8bce-caa8a8c4f44d }, OmicronPhysicalDiskConfig { id: 59c5a83e-c672-4165-9609-192aa1e34985, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }, pool_id: cee8396e-b813-42b3-9341-03d4b14507e0 }, OmicronPhysicalDiskConfig { id: d6b54216-ba9c-4665-bbfb-59c99d921052, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }, pool_id: 016894d7-b99a-44d7-bd91-820a9d7b1c8d }], generation: Generation(1) } file = sled-agent/src/rack_setup/service.rs:344 sled_address = [fd00:1122:3344:101::1]:12345
2142024-03-29T05:52:51.958ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:49204
2152024-03-29T05:52:51.958ZINFOSledAgent (StorageManager): Received OmicronPhysicalDisksEnsure { config: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, id: e108eec0-8a2a-4701-9cb4-4558ad340232, pool_id: ce966ff9-88a5-4653-bfe9-0d3a7a2ec9cc }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, id: 51e3dbcf-e17a-4662-a949-da2a639e7b5e, pool_id: 97f2d604-fcf7-497f-8f26-7e660b2d684b }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, id: 00216544-e869-44b2-95ee-cba360da3413, pool_id: 59af6774-3ad5-48ca-b81f-bfe9a37a0d07 }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, id: 5794b57f-31b0-4298-bbda-d8bf0042564d, pool_id: 78643ff1-d165-41d4-8bbd-429ec8539197 }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, id: a7713361-ccff-4308-af3b-3951fb7b4a1b, pool_id: cf82edba-c6a9-4917-87f1-2b21f22c3500 }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, id: ab4e69e3-de5d-45fc-8b01-5f8bc0833418, pool_id: eaa368be-d4d5-4952-8370-0d8467190389 }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }, id: d9ebd94c-1f2e-4b96-b7f5-2b50ac7ecb6b, pool_id: 2cb4ef32-43de-4203-8bce-caa8a8c4f44d }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }, id: 59c5a83e-c672-4165-9609-192aa1e34985, pool_id: cee8396e-b813-42b3-9341-03d4b14507e0 }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }, id: d6b54216-ba9c-4665-bbfb-59c99d921052, pool_id: 016894d7-b99a-44d7-bd91-820a9d7b1c8d }] }, tx: ... } file = sled-storage/src/manager.rs:381
2162024-03-29T05:52:51.958ZINFOSledAgent (StorageManager): No ledger in /pool/int/79ce42ff-9813-469e-9253-44dff191e19b/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2172024-03-29T05:52:51.958ZINFOSledAgent (StorageManager): No ledger in /pool/int/615b4df3-8040-4d04-947b-807c077fd24b/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2182024-03-29T05:52:51.958ZINFOSledAgent (StorageManager): No previously-stored 'requested disks', creating new ledger file = sled-storage/src/manager.rs:681 request = omicron_physical_disks_ensure
2192024-03-29T05:52:51.958ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:339
2202024-03-29T05:52:51.958ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
2212024-03-29T05:52:51.958ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
2222024-03-29T05:52:51.958ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:111 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } path = /scratch/u2_0.vdev
2232024-03-29T05:52:52.057ZWARNSledAgent: failed to register as a metric producer with Nexus error = "Metric registration error: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/metrics/producers): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/sled_agent.rs:1178 retry_after = 709.275531ms sled_id = 637eb488-60d7-4bc3-86a0-d192a08d81a4
2242024-03-29T05:52:52.064ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_0.vdev file = sled-hardware/src/disk.rs:301
2252024-03-29T05:52:52.064ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:307 id = ce966ff9-88a5-4653-bfe9-0d3a7a2ec9cc
2262024-03-29T05:52:52.701ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/637eb488-60d7-4bc3-86a0-d192a08d81a4): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 637eb488-60d7-4bc3-86a0-d192a08d81a4
2272024-03-29T05:52:52.913ZWARNSledAgent: failed to register as a metric producer with Nexus error = "Metric registration error: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/metrics/producers): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/sled_agent.rs:1178 retry_after = 641.842882ms sled_id = 637eb488-60d7-4bc3-86a0-d192a08d81a4
2282024-03-29T05:52:53.105ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: ce966ff9-88a5-4653-bfe9-0d3a7a2ec9cc, kind: External }
2292024-03-29T05:52:53.276ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
2302024-03-29T05:52:53.276ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
2312024-03-29T05:52:53.276ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
2322024-03-29T05:52:53.276ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
2332024-03-29T05:52:53.276ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2342024-03-29T05:52:53.276ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:38 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2352024-03-29T05:52:53.276ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
2362024-03-29T05:52:53.646ZWARNSledAgent: failed to register as a metric producer with Nexus error = "Metric registration error: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/metrics/producers): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/sled_agent.rs:1178 retry_after = 1.625599798s sled_id = 637eb488-60d7-4bc3-86a0-d192a08d81a4
2372024-03-29T05:52:53.878ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/637eb488-60d7-4bc3-86a0-d192a08d81a4): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 637eb488-60d7-4bc3-86a0-d192a08d81a4
2382024-03-29T05:52:53.878ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/637eb488-60d7-4bc3-86a0-d192a08d81a4): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 637eb488-60d7-4bc3-86a0-d192a08d81a4
2392024-03-29T05:52:53.880ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:57
2402024-03-29T05:52:54.085ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_ce966ff9-88a5-4653-bfe9-0d3a7a2ec9cc/crypt/zone file = sled-storage/src/dataset.rs:395
2412024-03-29T05:52:54.427ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_switch
2422024-03-29T05:52:55.172ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 39.863931ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2432024-03-29T05:52:55.174ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/637eb488-60d7-4bc3-86a0-d192a08d81a4): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 637eb488-60d7-4bc3-86a0-d192a08d81a4
2442024-03-29T05:52:55.733ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/637eb488-60d7-4bc3-86a0-d192a08d81a4): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 637eb488-60d7-4bc3-86a0-d192a08d81a4
2452024-03-29T05:52:55.733ZWARNSledAgent: failed to register as a metric producer with Nexus error = "Metric registration error: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/metrics/producers): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/sled_agent.rs:1178 retry_after = 3.850152462s sled_id = 637eb488-60d7-4bc3-86a0-d192a08d81a4
2462024-03-29T05:52:55.733ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 104.030767ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2472024-03-29T05:52:56.355ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 219.391576ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2482024-03-29T05:52:56.357ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/637eb488-60d7-4bc3-86a0-d192a08d81a4): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 637eb488-60d7-4bc3-86a0-d192a08d81a4
2492024-03-29T05:52:56.976ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 558.359549ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2502024-03-29T05:52:56.997ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_ce966ff9-88a5-4653-bfe9-0d3a7a2ec9cc file = sled-storage/src/dataset.rs:480
2512024-03-29T05:52:57.218ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
2522024-03-29T05:52:57.218ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
2532024-03-29T05:52:57.218ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
2542024-03-29T05:52:57.218ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
2552024-03-29T05:52:57.218ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:111 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } path = /scratch/u2_1.vdev
2562024-03-29T05:52:57.450ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_1.vdev file = sled-hardware/src/disk.rs:301
2572024-03-29T05:52:57.450ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:307 id = 97f2d604-fcf7-497f-8f26-7e660b2d684b
2582024-03-29T05:52:58.581ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 97f2d604-fcf7-497f-8f26-7e660b2d684b, kind: External }
2592024-03-29T05:52:58.839ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
2602024-03-29T05:52:58.839ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
2612024-03-29T05:52:58.839ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
2622024-03-29T05:52:58.839ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
2632024-03-29T05:52:58.839ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2642024-03-29T05:52:58.839ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:38 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2652024-03-29T05:52:58.839ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
2662024-03-29T05:52:59.337ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/637eb488-60d7-4bc3-86a0-d192a08d81a4): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 637eb488-60d7-4bc3-86a0-d192a08d81a4
2672024-03-29T05:52:59.337ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/637eb488-60d7-4bc3-86a0-d192a08d81a4): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 637eb488-60d7-4bc3-86a0-d192a08d81a4
2682024-03-29T05:52:59.594ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 558.908894ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2692024-03-29T05:52:59.597ZWARNSledAgent: failed to register as a metric producer with Nexus error = "Metric registration error: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/metrics/producers): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/sled_agent.rs:1178 retry_after = 10.957317303s sled_id = 637eb488-60d7-4bc3-86a0-d192a08d81a4
2702024-03-29T05:52:59.844ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:57
2712024-03-29T05:53:00.361ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_97f2d604-fcf7-497f-8f26-7e660b2d684b/crypt/zone file = sled-storage/src/dataset.rs:395
2722024-03-29T05:53:01.100ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/637eb488-60d7-4bc3-86a0-d192a08d81a4): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 637eb488-60d7-4bc3-86a0-d192a08d81a4
2732024-03-29T05:53:01.100ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/637eb488-60d7-4bc3-86a0-d192a08d81a4): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 637eb488-60d7-4bc3-86a0-d192a08d81a4
2742024-03-29T05:53:02.021ZINFOSledAgent (VmmReservoirManager): Finished setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:278 sled_id = 637eb488-60d7-4bc3-86a0-d192a08d81a4
2752024-03-29T05:53:02.031ZINFOSledAgent (ServiceManager): Ensuring tfportrear0_0/ll exists in zone file = sled-agent/src/services.rs:2286
2762024-03-29T05:53:02.104ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/637eb488-60d7-4bc3-86a0-d192a08d81a4): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 637eb488-60d7-4bc3-86a0-d192a08d81a4
2772024-03-29T05:53:02.114ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_97f2d604-fcf7-497f-8f26-7e660b2d684b file = sled-storage/src/dataset.rs:480
2782024-03-29T05:53:02.123ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
2792024-03-29T05:53:02.123ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
2802024-03-29T05:53:02.123ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
2812024-03-29T05:53:02.123ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
2822024-03-29T05:53:02.123ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:111 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } path = /scratch/u2_2.vdev
2832024-03-29T05:53:02.133ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_2.vdev file = sled-hardware/src/disk.rs:301
2842024-03-29T05:53:02.133ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:307 id = 59af6774-3ad5-48ca-b81f-bfe9a37a0d07
2852024-03-29T05:53:02.232ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 59af6774-3ad5-48ca-b81f-bfe9a37a0d07, kind: External }
2862024-03-29T05:53:02.241ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
2872024-03-29T05:53:02.241ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
2882024-03-29T05:53:02.241ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
2892024-03-29T05:53:02.241ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
2902024-03-29T05:53:02.241ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
2912024-03-29T05:53:02.241ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:38 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
2922024-03-29T05:53:02.241ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
2932024-03-29T05:53:02.278ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:57
2942024-03-29T05:53:02.286ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_59af6774-3ad5-48ca-b81f-bfe9a37a0d07/crypt/zone file = sled-storage/src/dataset.rs:395
2952024-03-29T05:53:02.409ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_59af6774-3ad5-48ca-b81f-bfe9a37a0d07 file = sled-storage/src/dataset.rs:480
2962024-03-29T05:53:02.417ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
2972024-03-29T05:53:02.417ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
2982024-03-29T05:53:02.417ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
2992024-03-29T05:53:02.417ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3002024-03-29T05:53:02.417ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:111 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } path = /scratch/u2_3.vdev
3012024-03-29T05:53:02.427ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_3.vdev file = sled-hardware/src/disk.rs:301
3022024-03-29T05:53:02.427ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:307 id = 78643ff1-d165-41d4-8bbd-429ec8539197
3032024-03-29T05:53:02.524ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 78643ff1-d165-41d4-8bbd-429ec8539197, kind: External }
3042024-03-29T05:53:02.532ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
3052024-03-29T05:53:02.532ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
3062024-03-29T05:53:02.532ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
3072024-03-29T05:53:02.532ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
3082024-03-29T05:53:02.532ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3092024-03-29T05:53:02.532ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:38 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3102024-03-29T05:53:02.532ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3112024-03-29T05:53:02.566ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:57
3122024-03-29T05:53:02.575ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_78643ff1-d165-41d4-8bbd-429ec8539197/crypt/zone file = sled-storage/src/dataset.rs:395
3132024-03-29T05:53:02.697ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_78643ff1-d165-41d4-8bbd-429ec8539197 file = sled-storage/src/dataset.rs:480
3142024-03-29T05:53:02.705ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
3152024-03-29T05:53:02.706ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
3162024-03-29T05:53:02.706ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
3172024-03-29T05:53:02.706ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3182024-03-29T05:53:02.706ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:111 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } path = /scratch/u2_4.vdev
3192024-03-29T05:53:02.715ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_4.vdev file = sled-hardware/src/disk.rs:301
3202024-03-29T05:53:02.715ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:307 id = cf82edba-c6a9-4917-87f1-2b21f22c3500
3212024-03-29T05:53:02.813ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: cf82edba-c6a9-4917-87f1-2b21f22c3500, kind: External }
3222024-03-29T05:53:02.822ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
3232024-03-29T05:53:02.822ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
3242024-03-29T05:53:02.822ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
3252024-03-29T05:53:02.822ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
3262024-03-29T05:53:02.822ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3272024-03-29T05:53:02.822ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:38 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3282024-03-29T05:53:02.822ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3292024-03-29T05:53:02.860ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:57
3302024-03-29T05:53:02.868ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_cf82edba-c6a9-4917-87f1-2b21f22c3500/crypt/zone file = sled-storage/src/dataset.rs:395
3312024-03-29T05:53:02.993ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_cf82edba-c6a9-4917-87f1-2b21f22c3500 file = sled-storage/src/dataset.rs:480
3322024-03-29T05:53:03.001ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
3332024-03-29T05:53:03.001ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
3342024-03-29T05:53:03.001ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
3352024-03-29T05:53:03.001ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3362024-03-29T05:53:03.001ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:111 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } path = /scratch/u2_5.vdev
3372024-03-29T05:53:03.011ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_5.vdev file = sled-hardware/src/disk.rs:301
3382024-03-29T05:53:03.011ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:307 id = eaa368be-d4d5-4952-8370-0d8467190389
3392024-03-29T05:53:03.107ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: eaa368be-d4d5-4952-8370-0d8467190389, kind: External }
3402024-03-29T05:53:03.115ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
3412024-03-29T05:53:03.115ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
3422024-03-29T05:53:03.115ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
3432024-03-29T05:53:03.115ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
3442024-03-29T05:53:03.115ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3452024-03-29T05:53:03.115ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:38 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3462024-03-29T05:53:03.115ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3472024-03-29T05:53:03.120ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/637eb488-60d7-4bc3-86a0-d192a08d81a4): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 637eb488-60d7-4bc3-86a0-d192a08d81a4
3482024-03-29T05:53:03.120ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/637eb488-60d7-4bc3-86a0-d192a08d81a4): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 637eb488-60d7-4bc3-86a0-d192a08d81a4
3492024-03-29T05:53:03.151ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:57
3502024-03-29T05:53:03.160ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_eaa368be-d4d5-4952-8370-0d8467190389/crypt/zone file = sled-storage/src/dataset.rs:395
3512024-03-29T05:53:03.286ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_eaa368be-d4d5-4952-8370-0d8467190389 file = sled-storage/src/dataset.rs:480
3522024-03-29T05:53:03.295ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
3532024-03-29T05:53:03.295ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
3542024-03-29T05:53:03.295ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
3552024-03-29T05:53:03.295ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3562024-03-29T05:53:03.295ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:111 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } path = /scratch/u2_6.vdev
3572024-03-29T05:53:03.305ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_6.vdev file = sled-hardware/src/disk.rs:301
3582024-03-29T05:53:03.305ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:307 id = 2cb4ef32-43de-4203-8bce-caa8a8c4f44d
3592024-03-29T05:53:03.401ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 2cb4ef32-43de-4203-8bce-caa8a8c4f44d, kind: External }
3602024-03-29T05:53:03.409ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
3612024-03-29T05:53:03.409ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
3622024-03-29T05:53:03.409ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
3632024-03-29T05:53:03.409ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
3642024-03-29T05:53:03.409ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3652024-03-29T05:53:03.409ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:38 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3662024-03-29T05:53:03.409ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3672024-03-29T05:53:03.444ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:57
3682024-03-29T05:53:03.452ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_2cb4ef32-43de-4203-8bce-caa8a8c4f44d/crypt/zone file = sled-storage/src/dataset.rs:395
3692024-03-29T05:53:03.575ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_2cb4ef32-43de-4203-8bce-caa8a8c4f44d file = sled-storage/src/dataset.rs:480
3702024-03-29T05:53:03.583ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
3712024-03-29T05:53:03.583ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
3722024-03-29T05:53:03.583ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
3732024-03-29T05:53:03.583ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3742024-03-29T05:53:03.583ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:111 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } path = /scratch/u2_7.vdev
3752024-03-29T05:53:03.593ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_7.vdev file = sled-hardware/src/disk.rs:301
3762024-03-29T05:53:03.593ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:307 id = cee8396e-b813-42b3-9341-03d4b14507e0
3772024-03-29T05:53:03.691ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: cee8396e-b813-42b3-9341-03d4b14507e0, kind: External }
3782024-03-29T05:53:03.699ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
3792024-03-29T05:53:03.699ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
3802024-03-29T05:53:03.699ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
3812024-03-29T05:53:03.699ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
3822024-03-29T05:53:03.699ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
3832024-03-29T05:53:03.699ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:38 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
3842024-03-29T05:53:03.699ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3852024-03-29T05:53:03.736ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:57
3862024-03-29T05:53:03.744ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_cee8396e-b813-42b3-9341-03d4b14507e0/crypt/zone file = sled-storage/src/dataset.rs:395
3872024-03-29T05:53:03.871ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_cee8396e-b813-42b3-9341-03d4b14507e0 file = sled-storage/src/dataset.rs:480
3882024-03-29T05:53:03.880ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
3892024-03-29T05:53:03.880ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
3902024-03-29T05:53:03.880ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
3912024-03-29T05:53:03.880ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3922024-03-29T05:53:03.880ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:111 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } path = /scratch/u2_8.vdev
3932024-03-29T05:53:03.889ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_8.vdev file = sled-hardware/src/disk.rs:301
3942024-03-29T05:53:03.889ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:307 id = 016894d7-b99a-44d7-bd91-820a9d7b1c8d
3952024-03-29T05:53:03.986ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 016894d7-b99a-44d7-bd91-820a9d7b1c8d, kind: External }
3962024-03-29T05:53:03.994ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
3972024-03-29T05:53:03.994ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
3982024-03-29T05:53:03.994ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
3992024-03-29T05:53:03.994ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
4002024-03-29T05:53:03.994ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4012024-03-29T05:53:03.994ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:38 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4022024-03-29T05:53:03.994ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
4032024-03-29T05:53:04.031ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:57
4042024-03-29T05:53:04.041ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_016894d7-b99a-44d7-bd91-820a9d7b1c8d/crypt/zone file = sled-storage/src/dataset.rs:395
4052024-03-29T05:53:04.122ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/637eb488-60d7-4bc3-86a0-d192a08d81a4): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 637eb488-60d7-4bc3-86a0-d192a08d81a4
4062024-03-29T05:53:04.174ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_016894d7-b99a-44d7-bd91-820a9d7b1c8d file = sled-storage/src/dataset.rs:480
4072024-03-29T05:53:04.182ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
4082024-03-29T05:53:04.183ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
4092024-03-29T05:53:04.183ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/79ce42ff-9813-469e-9253-44dff191e19b/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4102024-03-29T05:53:04.183ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 79ce42ff-9813-469e-9253-44dff191e19b, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 }, zpool_name: ZpoolName { id: ce966ff9-88a5-4653-bfe9-0d3a7a2ec9cc, kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 615b4df3-8040-4d04-947b-807c077fd24b, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 }, zpool_name: ZpoolName { id: 97f2d604-fcf7-497f-8f26-7e660b2d684b, kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 }, zpool_name: ZpoolName { id: 59af6774-3ad5-48ca-b81f-bfe9a37a0d07, kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 }, zpool_name: ZpoolName { id: 78643ff1-d165-41d4-8bbd-429ec8539197, kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, variant: U2, slot: 1030 }, zpool_name: ZpoolName { id: cf82edba-c6a9-4917-87f1-2b21f22c3500, kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, variant: U2, slot: 1031 }, zpool_name: ZpoolName { id: eaa368be-d4d5-4952-8370-0d8467190389, kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }, variant: U2, slot: 1032 }, zpool_name: ZpoolName { id: 2cb4ef32-43de-4203-8bce-caa8a8c4f44d, kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }, variant: U2, slot: 1033 }, zpool_name: ZpoolName { id: cee8396e-b813-42b3-9341-03d4b14507e0, kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }, variant: U2, slot: 1034 }, zpool_name: ZpoolName { id: 016894d7-b99a-44d7-bd91-820a9d7b1c8d, kind: External } }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
4112024-03-29T05:53:04.183ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/615b4df3-8040-4d04-947b-807c077fd24b/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4122024-03-29T05:53:04.183ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 12225133 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49204 req_id = bd85fe05-5de7-4001-9ba3-68077985074a response_code = 200 uri = /omicron-physical-disks
4132024-03-29T05:53:04.214ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(2), zones: [OmicronZoneConfig { id: ee405bc4-efc4-43d1-b97e-ce45b89dd237, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ce966ff9-88a5-4653-bfe9-0d3a7a2ec9cc, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: a2a66bc0-8fc4-40d7-8fa2-0f5af3ae231c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 97f2d604-fcf7-497f-8f26-7e660b2d684b, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 6a808e3b-83fa-48eb-8484-96eb4a45822d, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59af6774-3ad5-48ca-b81f-bfe9a37a0d07, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }] } file = sled-agent/src/rack_setup/service.rs:427 sled_address = [fd00:1122:3344:101::1]:12345
4142024-03-29T05:53:04.215ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:46495
4152024-03-29T05:53:04.215ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ee405bc4-efc4-43d1-b97e-ce45b89dd237, dataset_name: DatasetName { pool_name: ZpoolName { id: ce966ff9-88a5-4653-bfe9-0d3a7a2ec9cc, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
4162024-03-29T05:53:04.215ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ee405bc4-efc4-43d1-b97e-ce45b89dd237, dataset_name: DatasetName { pool_name: ZpoolName { id: ce966ff9-88a5-4653-bfe9-0d3a7a2ec9cc, kind: External }, kind: InternalDns }, responder: ... }
4172024-03-29T05:53:04.266ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a2a66bc0-8fc4-40d7-8fa2-0f5af3ae231c, dataset_name: DatasetName { pool_name: ZpoolName { id: 97f2d604-fcf7-497f-8f26-7e660b2d684b, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
4182024-03-29T05:53:04.266ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a2a66bc0-8fc4-40d7-8fa2-0f5af3ae231c, dataset_name: DatasetName { pool_name: ZpoolName { id: 97f2d604-fcf7-497f-8f26-7e660b2d684b, kind: External }, kind: InternalDns }, responder: ... }
4192024-03-29T05:53:04.317ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a808e3b-83fa-48eb-8484-96eb4a45822d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59af6774-3ad5-48ca-b81f-bfe9a37a0d07, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
4202024-03-29T05:53:04.317ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6a808e3b-83fa-48eb-8484-96eb4a45822d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59af6774-3ad5-48ca-b81f-bfe9a37a0d07, kind: External }, kind: InternalDns }, responder: ... }
4212024-03-29T05:53:04.368ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4222024-03-29T05:53:04.368ZINFOSledAgent (ServiceManager): No ledger in /pool/int/79ce42ff-9813-469e-9253-44dff191e19b/config/omicron-zones.json file = common/src/ledger.rs:188
4232024-03-29T05:53:04.368ZINFOSledAgent (ServiceManager): No ledger in /pool/int/615b4df3-8040-4d04-947b-807c077fd24b/config/omicron-zones.json file = common/src/ledger.rs:188
4242024-03-29T05:53:04.368ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4252024-03-29T05:53:04.416ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4262024-03-29T05:53:04.416ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4272024-03-29T05:53:04.416ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4282024-03-29T05:53:04.469ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_ee405bc4-efc4-43d1-b97e-ce45b89dd237 file = illumos-utils/src/zone.rs:315
4292024-03-29T05:53:04.472ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_a2a66bc0-8fc4-40d7-8fa2-0f5af3ae231c file = illumos-utils/src/zone.rs:315
4302024-03-29T05:53:04.476ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_6a808e3b-83fa-48eb-8484-96eb4a45822d file = illumos-utils/src/zone.rs:315
4312024-03-29T05:53:04.489ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_ee405bc4-efc4-43d1-b97e-ce45b89dd237 file = illumos-utils/src/zone.rs:354
4322024-03-29T05:53:04.494ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_a2a66bc0-8fc4-40d7-8fa2-0f5af3ae231c file = illumos-utils/src/zone.rs:354
4332024-03-29T05:53:04.498ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_6a808e3b-83fa-48eb-8484-96eb4a45822d file = illumos-utils/src/zone.rs:354
4342024-03-29T05:53:05.119ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/637eb488-60d7-4bc3-86a0-d192a08d81a4): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 637eb488-60d7-4bc3-86a0-d192a08d81a4
4352024-03-29T05:53:05.125ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/637eb488-60d7-4bc3-86a0-d192a08d81a4): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 637eb488-60d7-4bc3-86a0-d192a08d81a4
4362024-03-29T05:53:06.077ZINFOSledAgent (ServiceManager): Ensuring bootstrap address fdb0:200::2 exists in switch zone file = sled-agent/src/services.rs:2303
4372024-03-29T05:53:06.077ZINFOSledAgent (ServiceManager): Adding bootstrap address file = illumos-utils/src/running_zone.rs:626 zone = oxz_switch
4382024-03-29T05:53:06.127ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/637eb488-60d7-4bc3-86a0-d192a08d81a4): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 637eb488-60d7-4bc3-86a0-d192a08d81a4
4392024-03-29T05:53:06.799ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_ee405bc4-efc4-43d1-b97e-ce45b89dd237: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService1"/> <propval type="astring" name="gateway" value="fd00:1122:3344:1::2"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:1::1"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:1::1]:5353"/> <propval type="astring" name="dns_address" value="[fd00:1122:3344:1::1]:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4402024-03-29T05:53:06.800ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:87 prefix = Ipv6Prefix { addr: fd00:1122:3344:1::, len: 64 }
4412024-03-29T05:53:06.800ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_internal_dns_ee405bc4-efc4-43d1-b97e-ce45b89dd237
4422024-03-29T05:53:07.288ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_a2a66bc0-8fc4-40d7-8fa2-0f5af3ae231c: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService2"/> <propval type="astring" name="gateway" value="fd00:1122:3344:2::2"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:2::1"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:2::1]:5353"/> <propval type="astring" name="dns_address" value="[fd00:1122:3344:2::1]:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4432024-03-29T05:53:07.769ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:87 prefix = Ipv6Prefix { addr: fd00:1122:3344:2::, len: 64 }
4442024-03-29T05:53:07.769ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_6a808e3b-83fa-48eb-8484-96eb4a45822d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService3"/> <propval type="astring" name="gateway" value="fd00:1122:3344:3::2"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:3::1"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:3::1]:5353"/> <propval type="astring" name="dns_address" value="[fd00:1122:3344:3::1]:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4452024-03-29T05:53:07.769ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:87 prefix = Ipv6Prefix { addr: fd00:1122:3344:3::, len: 64 }
4462024-03-29T05:53:07.780ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_ee405bc4-efc4-43d1-b97e-ce45b89dd237") failed: Property not found. retry in 43.953222ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_ee405bc4-efc4-43d1-b97e-ce45b89dd237
4472024-03-29T05:53:07.780ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_internal_dns_a2a66bc0-8fc4-40d7-8fa2-0f5af3ae231c
4482024-03-29T05:53:07.784ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_internal_dns_6a808e3b-83fa-48eb-8484-96eb4a45822d
4492024-03-29T05:53:07.833ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_ee405bc4-efc4-43d1-b97e-ce45b89dd237") failed: Property not found. retry in 106.528039ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_ee405bc4-efc4-43d1-b97e-ce45b89dd237
4502024-03-29T05:53:07.910ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6a808e3b-83fa-48eb-8484-96eb4a45822d") failed: Property not found. retry in 37.541619ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6a808e3b-83fa-48eb-8484-96eb4a45822d
4512024-03-29T05:53:07.916ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_a2a66bc0-8fc4-40d7-8fa2-0f5af3ae231c") failed: Property not found. retry in 35.871746ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_a2a66bc0-8fc4-40d7-8fa2-0f5af3ae231c
4522024-03-29T05:53:07.947ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_ee405bc4-efc4-43d1-b97e-ce45b89dd237") failed: Property not found. retry in 124.503578ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_ee405bc4-efc4-43d1-b97e-ce45b89dd237
4532024-03-29T05:53:07.957ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6a808e3b-83fa-48eb-8484-96eb4a45822d") failed: Property not found. retry in 139.688944ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6a808e3b-83fa-48eb-8484-96eb4a45822d
4542024-03-29T05:53:07.964ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_a2a66bc0-8fc4-40d7-8fa2-0f5af3ae231c") failed: Property not found. retry in 135.679855ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_a2a66bc0-8fc4-40d7-8fa2-0f5af3ae231c
4552024-03-29T05:53:08.110ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_ee405bc4-efc4-43d1-b97e-ce45b89dd237") failed: Property not found. retry in 247.481114ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_ee405bc4-efc4-43d1-b97e-ce45b89dd237
4562024-03-29T05:53:08.125ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_a2a66bc0-8fc4-40d7-8fa2-0f5af3ae231c") failed: Property not found. retry in 233.786608ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_a2a66bc0-8fc4-40d7-8fa2-0f5af3ae231c
4572024-03-29T05:53:08.178ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6a808e3b-83fa-48eb-8484-96eb4a45822d") failed: Property not found. retry in 224.016779ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6a808e3b-83fa-48eb-8484-96eb4a45822d
4582024-03-29T05:53:08.383ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_ee405bc4-efc4-43d1-b97e-ce45b89dd237") failed: Property not found. retry in 1.008420652s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_ee405bc4-efc4-43d1-b97e-ce45b89dd237
4592024-03-29T05:53:08.423ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_a2a66bc0-8fc4-40d7-8fa2-0f5af3ae231c") failed: Property not found. retry in 439.871577ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_a2a66bc0-8fc4-40d7-8fa2-0f5af3ae231c
4602024-03-29T05:53:08.435ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6a808e3b-83fa-48eb-8484-96eb4a45822d") failed: Property not found. retry in 558.599633ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6a808e3b-83fa-48eb-8484-96eb4a45822d
4612024-03-29T05:53:08.872ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_a2a66bc0-8fc4-40d7-8fa2-0f5af3ae231c") failed: Property not found. retry in 595.743891ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_a2a66bc0-8fc4-40d7-8fa2-0f5af3ae231c
4622024-03-29T05:53:09.005ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6a808e3b-83fa-48eb-8484-96eb4a45822d") failed: Property not found. retry in 704.169727ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6a808e3b-83fa-48eb-8484-96eb4a45822d
4632024-03-29T05:53:09.400ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_ee405bc4-efc4-43d1-b97e-ce45b89dd237") failed: Property not found. retry in 1.311350037s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_ee405bc4-efc4-43d1-b97e-ce45b89dd237
4642024-03-29T05:53:09.480ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_a2a66bc0-8fc4-40d7-8fa2-0f5af3ae231c") failed: Property not found. retry in 515.920861ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_a2a66bc0-8fc4-40d7-8fa2-0f5af3ae231c
4652024-03-29T05:53:09.721ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6a808e3b-83fa-48eb-8484-96eb4a45822d") failed: Property not found. retry in 1.389372011s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6a808e3b-83fa-48eb-8484-96eb4a45822d
4662024-03-29T05:53:10.006ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_a2a66bc0-8fc4-40d7-8fa2-0f5af3ae231c") failed: Property not found. retry in 1.148704336s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_a2a66bc0-8fc4-40d7-8fa2-0f5af3ae231c
4672024-03-29T05:53:10.627ZINFOSledAgent (ServiceManager): Forwarding bootstrap traffic via oxBootstrap0 to fe80::8:20ff:fe83:1a9f file = sled-agent/src/services.rs:2310
4682024-03-29T05:53:10.632ZINFOSledAgent (ServiceManager): Zone not using gateway (even though underlay is up) file = sled-agent/src/services.rs:2371
4692024-03-29T05:53:10.720ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_ee405bc4-efc4-43d1-b97e-ce45b89dd237") failed: Property not found. retry in 902.583206ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_ee405bc4-efc4-43d1-b97e-ce45b89dd237
4702024-03-29T05:53:10.752ZINFOSledAgent (ServiceManager): Setting up dendrite service file = sled-agent/src/services.rs:2556
4712024-03-29T05:53:10.995ZINFOSledAgent (ServiceManager): Setting up lldpd service file = sled-agent/src/services.rs:2720
4722024-03-29T05:53:11.119ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6a808e3b-83fa-48eb-8484-96eb4a45822d") failed: Property not found. retry in 1.057766573s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6a808e3b-83fa-48eb-8484-96eb4a45822d
4732024-03-29T05:53:11.163ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_a2a66bc0-8fc4-40d7-8fa2-0f5af3ae231c") failed: Property not found. retry in 1.321828395s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_a2a66bc0-8fc4-40d7-8fa2-0f5af3ae231c
4742024-03-29T05:53:11.184ZINFOSledAgent (ServiceManager): Setting up MGS service file = sled-agent/src/services.rs:2435
4752024-03-29T05:53:11.483ZINFOSledAgent (ServiceManager): Setting up wicketd service file = sled-agent/src/services.rs:2470
4762024-03-29T05:53:11.631ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_ee405bc4-efc4-43d1-b97e-ce45b89dd237") failed: Property not found. retry in 914.094221ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_ee405bc4-efc4-43d1-b97e-ce45b89dd237
4772024-03-29T05:53:11.691ZINFOSledAgent (ServiceManager): Setting up mgd service file = sled-agent/src/services.rs:2758
4782024-03-29T05:53:11.873ZINFOSledAgent (ServiceManager): Setting up mg-ddm service file = sled-agent/src/services.rs:2785
4792024-03-29T05:53:12.088ZINFOSledAgent (ServiceManager): Setting up tfport service file = sled-agent/src/services.rs:2664
4802024-03-29T05:53:12.186ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6a808e3b-83fa-48eb-8484-96eb4a45822d") failed: Property not found. retry in 1.009799845s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6a808e3b-83fa-48eb-8484-96eb4a45822d
4812024-03-29T05:53:12.273ZINFOSledAgent (ServiceManager): Setting up Simulated SP service file = sled-agent/src/services.rs:2464
4822024-03-29T05:53:12.283ZINFOSledAgent (ServiceManager): Re-enabling running switch zone (new address) file = sled-agent/src/services.rs:3843 new = [fd00:1122:3344:101::2, ::1] old = [::1]
4832024-03-29T05:53:12.283ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists file = sled-agent/src/services.rs:3859
4842024-03-29T05:53:12.283ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::2, prefix: 64 })) file = illumos-utils/src/running_zone.rs:609 zone = oxz_switch
4852024-03-29T05:53:12.495ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_a2a66bc0-8fc4-40d7-8fa2-0f5af3ae231c") failed: Property not found. retry in 1.278026137s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_a2a66bc0-8fc4-40d7-8fa2-0f5af3ae231c
4862024-03-29T05:53:12.836ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists - OK file = sled-agent/src/services.rs:3867
4872024-03-29T05:53:12.842ZINFOSledAgent (ServiceManager): configuring dendrite service file = sled-agent/src/services.rs:3923
4882024-03-29T05:53:12.987ZINFOSledAgent (ServiceManager): configuring lldp service file = sled-agent/src/services.rs:3985
4892024-03-29T05:53:13.092ZINFOSledAgent (ServiceManager): configuring wicketd file = sled-agent/src/services.rs:3966 rack_subnet = fd00:1122:3344::
4902024-03-29T05:53:13.116ZINFOSledAgent (ServiceManager): configuring mgd service file = sled-agent/src/services.rs:4008
4912024-03-29T05:53:13.222ZINFOSledAgent (ServiceManager): configuring mg-ddm service file = sled-agent/src/services.rs:4035
4922024-03-29T05:53:13.346ZINFOSledAgent (ServiceManager): Determining physical location of our switch zone at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:364
4932024-03-29T05:53:13.379ZINFOSledAgent (ServiceManager): Initializing 1 Uplinks on Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:412
4942024-03-29T05:53:13.411ZINFOSledAgent (ServiceManager): Building Port Configuration file = sled-agent/src/bootstrap/early_networking.rs:589
4952024-03-29T05:53:13.411ZINFOSledAgent (ServiceManager): Checking dendrite uptime file = sled-agent/src/bootstrap/early_networking.rs:627
4962024-03-29T05:53:13.411ZINFOSledAgent (ServiceManager): Dendrite online file = sled-agent/src/bootstrap/early_networking.rs:630 uptime = 3
4972024-03-29T05:53:13.411ZINFOSledAgent (ServiceManager): Configuring default uplink on switch config = PortSettings {\n links: {\n "0": LinkSettings {\n addrs: [\n 10.151.1.174,\n ],\n params: LinkCreate {\n autoneg: false,\n fec: None,\n kr: false,\n lane: Some(\n LinkId(\n 0,\n ),\n ),\n speed: Speed40G,\n },\n },\n },\n} file = sled-agent/src/bootstrap/early_networking.rs:434
4982024-03-29T05:53:13.490ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:381
4992024-03-29T05:53:13.491ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }
5002024-03-29T05:53:13.491ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }
5012024-03-29T05:53:13.491ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }
5022024-03-29T05:53:13.491ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }
5032024-03-29T05:53:13.491ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }
5042024-03-29T05:53:13.491ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }
5052024-03-29T05:53:13.491ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }
5062024-03-29T05:53:13.491ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }
5072024-03-29T05:53:13.491ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }
5082024-03-29T05:53:13.491ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }
5092024-03-29T05:53:13.491ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }
5102024-03-29T05:53:13.796ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2997 zone = oxz_internal_dns_ee405bc4-efc4-43d1-b97e-ce45b89dd237
5112024-03-29T05:53:13.796ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2997 zone = oxz_internal_dns_a2a66bc0-8fc4-40d7-8fa2-0f5af3ae231c
5122024-03-29T05:53:13.796ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2997 zone = oxz_internal_dns_6a808e3b-83fa-48eb-8484-96eb4a45822d
5132024-03-29T05:53:13.796ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/79ce42ff-9813-469e-9253-44dff191e19b/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5142024-03-29T05:53:13.796ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/615b4df3-8040-4d04-947b-807c077fd24b/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5152024-03-29T05:53:13.796ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 9581712 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46495 req_id = 6d506a78-2d9b-4baa-bd4e-ad8dd6803775 response_code = 204 uri = /omicron-zones
5162024-03-29T05:53:13.798ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:1::1]:5353 file = sled-agent/src/rack_setup/service.rs:545
5172024-03-29T05:53:13.874ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:2::1]:5353 file = sled-agent/src/rack_setup/service.rs:545
5182024-03-29T05:53:14.336ZWARNSledAgent (RSS): failed to write DNS configuration (will retry in 128.667661ms) dns_config_addr = [fd00:1122:3344:2::1]:5353 error_message = Communication Error: error sending request for url (http://[fd00:1122:3344:2::1]:5353/config): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:565
5192024-03-29T05:53:14.506ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:3::1]:5353 file = sled-agent/src/rack_setup/service.rs:545
5202024-03-29T05:53:14.571ZINFOSledAgent (RSS): Configured all DNS servers file = sled-agent/src/rack_setup/service.rs:581
5212024-03-29T05:53:14.571ZINFOSledAgent (RSS): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:258
5222024-03-29T05:53:14.603ZINFOSledAgent (RSS): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:275
5232024-03-29T05:53:14.605ZINFOSledAgent (RSS): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:300
5242024-03-29T05:53:14.605ZINFOSledAgent (RSS): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:192
5252024-03-29T05:53:14.634ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(3), zones: [OmicronZoneConfig { id: ee405bc4-efc4-43d1-b97e-ce45b89dd237, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ce966ff9-88a5-4653-bfe9-0d3a7a2ec9cc, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: a2a66bc0-8fc4-40d7-8fa2-0f5af3ae231c, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 97f2d604-fcf7-497f-8f26-7e660b2d684b, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 6a808e3b-83fa-48eb-8484-96eb4a45822d, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 59af6774-3ad5-48ca-b81f-bfe9a37a0d07, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: c606920f-6a36-4d68-be6e-e371f2cb6a6c, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6a28132f-9760-48d5-9c81-896b2e51354b, kind: Service { id: c606920f-6a36-4d68-be6e-e371f2cb6a6c }, name: Name("ntp-c606920f-6a36-4d68-be6e-e371f2cb6a6c"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 191, 144])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }] } file = sled-agent/src/rack_setup/service.rs:427 sled_address = [fd00:1122:3344:101::1]:12345
5262024-03-29T05:53:14.634ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:62914
5272024-03-29T05:53:14.634ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ee405bc4-efc4-43d1-b97e-ce45b89dd237, dataset_name: DatasetName { pool_name: ZpoolName { id: ce966ff9-88a5-4653-bfe9-0d3a7a2ec9cc, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
5282024-03-29T05:53:14.634ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ee405bc4-efc4-43d1-b97e-ce45b89dd237, dataset_name: DatasetName { pool_name: ZpoolName { id: ce966ff9-88a5-4653-bfe9-0d3a7a2ec9cc, kind: External }, kind: InternalDns }, responder: ... }
5292024-03-29T05:53:14.656ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a2a66bc0-8fc4-40d7-8fa2-0f5af3ae231c, dataset_name: DatasetName { pool_name: ZpoolName { id: 97f2d604-fcf7-497f-8f26-7e660b2d684b, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
5302024-03-29T05:53:14.657ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a2a66bc0-8fc4-40d7-8fa2-0f5af3ae231c, dataset_name: DatasetName { pool_name: ZpoolName { id: 97f2d604-fcf7-497f-8f26-7e660b2d684b, kind: External }, kind: InternalDns }, responder: ... }
5312024-03-29T05:53:14.675ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a808e3b-83fa-48eb-8484-96eb4a45822d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59af6774-3ad5-48ca-b81f-bfe9a37a0d07, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
5322024-03-29T05:53:14.675ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6a808e3b-83fa-48eb-8484-96eb4a45822d, dataset_name: DatasetName { pool_name: ZpoolName { id: 59af6774-3ad5-48ca-b81f-bfe9a37a0d07, kind: External }, kind: InternalDns }, responder: ... }
5332024-03-29T05:53:14.695ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
5342024-03-29T05:53:14.695ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/79ce42ff-9813-469e-9253-44dff191e19b/config/omicron-zones.json file = common/src/ledger.rs:177
5352024-03-29T05:53:14.695ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/615b4df3-8040-4d04-947b-807c077fd24b/config/omicron-zones.json file = common/src/ledger.rs:177
5362024-03-29T05:53:14.695ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
5372024-03-29T05:53:14.706ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:258
5382024-03-29T05:53:14.738ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:275
5392024-03-29T05:53:14.739ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:300
5402024-03-29T05:53:14.739ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:192
5412024-03-29T05:53:14.784ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:394 port = Port { inner: PortInner { name: "opte0", ip: 172.30.3.5, mac: MacAddr6([168, 64, 37, 255, 191, 144]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.3.1 }, vnic: "vopte0" } }
5422024-03-29T05:53:14.784ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1277 zone_type = ntp
5432024-03-29T05:53:14.785ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.1.169, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
5442024-03-29T05:53:14.785ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.1.169, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
5452024-03-29T05:53:14.785ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.1.169, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
5462024-03-29T05:53:14.786ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
5472024-03-29T05:53:14.811ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_ntp_c606920f-6a36-4d68-be6e-e371f2cb6a6c file = illumos-utils/src/zone.rs:315
5482024-03-29T05:53:14.830ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_ntp_c606920f-6a36-4d68-be6e-e371f2cb6a6c file = illumos-utils/src/zone.rs:354
5492024-03-29T05:53:16.032ZINFOSledAgent (ServiceManager): Profile for oxz_ntp_c606920f-6a36-4d68-be6e-e371f2cb6a6c: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService4"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::1b"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="1.1.1.1" /> <value_node value="9.9.9.9" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/ntp"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="allow" value="fd00:1122:3344:100::/56"/> <propval type="boolean" name="boundary" value="true"/> <propval type="astring" name="server" value="0.pool.ntp.org"/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte0"/> <propval type="astring" name="gateway" value="172.30.3.1"/> <propval type="astring" name="ip" value="172.30.3.5"/> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
5502024-03-29T05:53:16.032ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_ntp_c606920f-6a36-4d68-be6e-e371f2cb6a6c
5512024-03-29T05:53:16.175ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_c606920f-6a36-4d68-be6e-e371f2cb6a6c") failed: Property not found. retry in 39.04155ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_c606920f-6a36-4d68-be6e-e371f2cb6a6c
5522024-03-29T05:53:16.222ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_c606920f-6a36-4d68-be6e-e371f2cb6a6c") failed: Property not found. retry in 132.331296ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_c606920f-6a36-4d68-be6e-e371f2cb6a6c
5532024-03-29T05:53:16.391ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_c606920f-6a36-4d68-be6e-e371f2cb6a6c") failed: Property not found. retry in 174.983907ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_c606920f-6a36-4d68-be6e-e371f2cb6a6c
5542024-03-29T05:53:16.574ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_c606920f-6a36-4d68-be6e-e371f2cb6a6c") failed: Property not found. retry in 562.88496ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_c606920f-6a36-4d68-be6e-e371f2cb6a6c
5552024-03-29T05:53:16.770ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:56974
5562024-03-29T05:53:16.770ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 96 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:56974 req_id = b3f505f0-cece-4bc1-8078-32b22e2b713c response_code = 200 uri = /baseboard
5572024-03-29T05:53:17.148ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_c606920f-6a36-4d68-be6e-e371f2cb6a6c") failed: Property not found. retry in 683.870797ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_c606920f-6a36-4d68-be6e-e371f2cb6a6c
5582024-03-29T05:53:17.842ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_c606920f-6a36-4d68-be6e-e371f2cb6a6c") failed: Property not found. retry in 1.231518453s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_c606920f-6a36-4d68-be6e-e371f2cb6a6c
5592024-03-29T05:53:19.083ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_c606920f-6a36-4d68-be6e-e371f2cb6a6c") failed: Property not found. retry in 997.278106ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_c606920f-6a36-4d68-be6e-e371f2cb6a6c
5602024-03-29T05:53:20.089ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_c606920f-6a36-4d68-be6e-e371f2cb6a6c") failed: Property not found. retry in 976.827125ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_c606920f-6a36-4d68-be6e-e371f2cb6a6c
5612024-03-29T05:53:21.075ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_c606920f-6a36-4d68-be6e-e371f2cb6a6c") failed: Property not found. retry in 1.012469133s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_c606920f-6a36-4d68-be6e-e371f2cb6a6c
5622024-03-29T05:53:22.109ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2997 zone = oxz_ntp_c606920f-6a36-4d68-be6e-e371f2cb6a6c
5632024-03-29T05:53:22.109ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/79ce42ff-9813-469e-9253-44dff191e19b/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5642024-03-29T05:53:22.110ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/615b4df3-8040-4d04-947b-807c077fd24b/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5652024-03-29T05:53:22.110ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 7475738 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:62914 req_id = 3bb5578e-7fd6-44fe-b087-3fd3f35fb05c response_code = 204 uri = /omicron-zones
5662024-03-29T05:53:22.112ZINFOSledAgent (RSS): Waiting for rack time synchronization file = sled-agent/src/rack_setup/service.rs:622
5672024-03-29T05:53:22.141ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
5682024-03-29T05:53:22.141ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:41077
5692024-03-29T05:53:22.150ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_c606920f-6a36-4d68-be6e-e371f2cb6a6c': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3574
5702024-03-29T05:53:22.150ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 9274 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:41077 req_id = 0b3b1e95-6cbb-41a5-beac-afc378e4ce4b response_code = 500 uri = /timesync
5712024-03-29T05:53:22.154ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
5722024-03-29T05:53:22.320ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
5732024-03-29T05:53:22.320ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:65520
5742024-03-29T05:53:26.329ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_c606920f-6a36-4d68-be6e-e371f2cb6a6c': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3574
5752024-03-29T05:53:26.329ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 4009180 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:65520 req_id = 02f8bf50-9efc-4782-9dd0-b03ea6b73cf0 response_code = 500 uri = /timesync
5762024-03-29T05:53:26.332ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
5772024-03-29T05:53:26.492ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
5782024-03-29T05:53:26.492ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:63089
5792024-03-29T05:53:26.501ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_c606920f-6a36-4d68-be6e-e371f2cb6a6c': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3574
5802024-03-29T05:53:26.502ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 8970 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:63089 req_id = 69490f5b-58fe-40f9-8711-4d5e36301497 response_code = 500 uri = /timesync
5812024-03-29T05:53:26.505ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
5822024-03-29T05:53:26.707ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
5832024-03-29T05:53:26.707ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:40468
5842024-03-29T05:53:26.716ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_c606920f-6a36-4d68-be6e-e371f2cb6a6c': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3574
5852024-03-29T05:53:26.716ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 8893 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:40468 req_id = 22123811-cb87-40c0-bfbe-592320ad0adc response_code = 500 uri = /timesync
5862024-03-29T05:53:26.719ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
5872024-03-29T05:53:26.908ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
5882024-03-29T05:53:26.908ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:41762
5892024-03-29T05:53:26.917ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 8133 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:41762 req_id = 2cdd4219-8266-490b-9d45-a14a9aa8ecdc response_code = 200 uri = /timesync
5902024-03-29T05:53:26.919ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711691605.2858346, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
5912024-03-29T05:53:26.919ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
5922024-03-29T05:53:27.137ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
5932024-03-29T05:53:27.137ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:45534
5942024-03-29T05:53:27.145ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 7886 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:45534 req_id = 31512624-9a39-4d71-b111-bb824428e5a3 response_code = 200 uri = /timesync
5952024-03-29T05:53:27.147ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711691605.2858346, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
5962024-03-29T05:53:27.147ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
5972024-03-29T05:53:27.538ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
5982024-03-29T05:53:27.538ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:65033
5992024-03-29T05:53:27.547ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 8276 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:65033 req_id = 447d0946-6948-46c3-9845-bf47d8ca5b90 response_code = 200 uri = /timesync
6002024-03-29T05:53:27.549ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711691605.2858346, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
6012024-03-29T05:53:27.549ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
6022024-03-29T05:53:27.908ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
6032024-03-29T05:53:27.908ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:50714
6042024-03-29T05:53:27.916ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 8023 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:50714 req_id = 68e5fe99-f472-4fa9-bfe0-56a43d5d00a2 response_code = 200 uri = /timesync
6052024-03-29T05:53:27.918ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711691605.2858346, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
6062024-03-29T05:53:27.918ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
6072024-03-29T05:53:28.408ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
6082024-03-29T05:53:28.408ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:48421
6092024-03-29T05:53:28.417ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 8564 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:48421 req_id = b587b92c-3cdf-498f-bc5c-f9e28de77430 response_code = 200 uri = /timesync
6102024-03-29T05:53:28.419ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711691605.2858346, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
6112024-03-29T05:53:28.419ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
6122024-03-29T05:53:28.735ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
6132024-03-29T05:53:28.735ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:39894
6142024-03-29T05:53:28.744ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 9227 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:39894 req_id = 39e44a3b-1cad-492c-a19a-8594f0f329b9 response_code = 200 uri = /timesync
6152024-03-29T05:53:28.746ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711691605.2858346, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
6162024-03-29T05:53:28.746ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
6172024-03-29T05:53:29.037ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
6182024-03-29T05:53:29.037ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:49380
6192024-03-29T05:53:29.046ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 8542 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:49380 req_id = c2be4de6-27c0-4280-9658-e11888ac528e response_code = 200 uri = /timesync
6202024-03-29T05:53:29.049ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711691605.2858346, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
6212024-03-29T05:53:29.049ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
6222024-03-29T05:53:29.598ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
6232024-03-29T05:53:29.598ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42264
6242024-03-29T05:53:29.608ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 9402 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:42264 req_id = e86c32f5-c1e8-4ea0-9e0d-70bf55a54ddf response_code = 200 uri = /timesync
6252024-03-29T05:53:29.611ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711691605.2858346, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
6262024-03-29T05:53:29.611ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
6272024-03-29T05:53:30.161ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
6282024-03-29T05:53:30.161ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:51329
6292024-03-29T05:53:30.169ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 7727 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:51329 req_id = 148d3a95-04fb-40e0-9102-87240a59d597 response_code = 200 uri = /timesync
6302024-03-29T05:53:30.171ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711691605.2858346, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
6312024-03-29T05:53:30.171ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
6322024-03-29T05:53:31.539ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
6332024-03-29T05:53:31.539ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:37028
6342024-03-29T05:53:31.548ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 8472 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:37028 req_id = 6f99a446-9390-42dc-9a83-6bc15d8a5715 response_code = 200 uri = /timesync
6352024-03-29T05:53:31.550ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711691605.2858346, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
6362024-03-29T05:53:31.550ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
6372024-03-29T05:53:33.039ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
6382024-03-29T05:53:33.040ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:53918
6392024-03-29T05:53:33.049ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 8848 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:53918 req_id = 25af27a8-f648-468d-a538-e6a420d61c35 response_code = 200 uri = /timesync
6402024-03-29T05:53:33.052ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711691605.2858346, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
6412024-03-29T05:53:33.052ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
6422024-03-29T05:53:33.851ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
6432024-03-29T05:53:33.851ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:48417
6442024-03-29T05:53:33.860ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 8988 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:48417 req_id = ff81bb74-7705-4a97-9c08-1863db311f0e response_code = 200 uri = /timesync
6452024-03-29T05:53:33.864ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711691605.2858346, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
6462024-03-29T05:53:33.864ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
6472024-03-29T05:53:34.891ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
6482024-03-29T05:53:34.891ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:41380
6492024-03-29T05:53:34.899ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 8392 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:41380 req_id = a07f4d63-75a2-44ae-95fd-858c99da7c98 response_code = 200 uri = /timesync
6502024-03-29T05:53:34.901ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711691605.2858346, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
6512024-03-29T05:53:34.901ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
6522024-03-29T05:53:36.036ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
6532024-03-29T05:53:36.036ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:37244
6542024-03-29T05:53:36.044ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 8363 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:37244 req_id = f1a04d5e-fac6-48d5-9bb5-d23e1b45ded1 response_code = 200 uri = /timesync
6552024-03-29T05:53:36.047ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711691605.2858346, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
6562024-03-29T05:53:36.047ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
6572024-03-29T05:53:37.580ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
6582024-03-29T05:53:37.580ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:33060
6592024-03-29T05:53:37.589ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 8372 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:33060 req_id = e24a34d0-c6a7-40a1-a950-ae21045499fb response_code = 200 uri = /timesync
6602024-03-29T05:53:37.591ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711691605.2858346, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
6612024-03-29T05:53:37.591ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
6622024-03-29T05:53:39.728ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
6632024-03-29T05:53:39.728ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:53015
6642024-03-29T05:53:39.736ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 7945 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:53015 req_id = ac42a780-9202-4d68-8d9f-9d3976505feb response_code = 200 uri = /timesync
6652024-03-29T05:53:39.738ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711691605.2858346, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
6662024-03-29T05:53:39.738ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
6672024-03-29T05:53:42.998ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
6682024-03-29T05:53:42.998ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:55793
6692024-03-29T05:53:43.006ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 7776 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:55793 req_id = 0b66c476-1714-4743-a966-fb1eb6ec3da1 response_code = 200 uri = /timesync
6702024-03-29T05:53:43.008ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711691605.2858346, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
6712024-03-29T05:53:43.008ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
6722024-03-29T05:53:45.903ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
6732024-03-29T05:53:45.903ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:63499
6742024-03-29T05:53:45.912ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 8818 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:63499 req_id = ff6ce4c0-79a1-431f-8ef4-6fc221ca8b88 response_code = 200 uri = /timesync
6752024-03-29T05:53:45.914ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711691605.2858346, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
6762024-03-29T05:53:45.914ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
6772024-03-29T05:53:46.802ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:62707
6782024-03-29T05:53:46.803ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 872 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:62707 req_id = 5024d211-36d0-4da7-bf0c-4ad87def928b response_code = 200 uri = /baseboard
6792024-03-29T05:53:49.759ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
6802024-03-29T05:53:49.759ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:57345
6812024-03-29T05:53:49.767ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 8175 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:57345 req_id = 58c5c9b4-9256-4d7c-aa20-b24f2213d62d response_code = 200 uri = /timesync
6822024-03-29T05:53:49.769ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711691605.2858346, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
6832024-03-29T05:53:49.769ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
6842024-03-29T05:53:53.539ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
6852024-03-29T05:53:53.539ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:33740
6862024-03-29T05:53:53.548ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 8517 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:33740 req_id = 63b24068-affd-4f64-8fa0-5285e328b5ed response_code = 200 uri = /timesync
6872024-03-29T05:53:53.550ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711691605.2858346, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
6882024-03-29T05:53:53.550ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
6892024-03-29T05:53:58.345ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
6902024-03-29T05:53:58.345ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:38225
6912024-03-29T05:53:58.354ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 9138 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:38225 req_id = 20ad8868-22e9-44c7-b60f-8d132f7a452b response_code = 200 uri = /timesync
6922024-03-29T05:53:58.357ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711691605.2858346, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
6932024-03-29T05:53:58.357ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
6942024-03-29T05:54:02.988ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
6952024-03-29T05:54:02.988ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:54311
6962024-03-29T05:54:02.997ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 8760 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:54311 req_id = 2b59d2b4-8589-4963-b77f-67eede8f6971 response_code = 200 uri = /timesync
6972024-03-29T05:54:02.999ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711691605.2858346, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
6982024-03-29T05:54:02.999ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
6992024-03-29T05:54:10.255ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
7002024-03-29T05:54:10.255ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:33536
7012024-03-29T05:54:10.264ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 8529 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:33536 req_id = 49ff77e5-fb45-4cc7-b8ca-8ad655119e59 response_code = 200 uri = /timesync
7022024-03-29T05:54:10.266ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711691605.2858346, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
7032024-03-29T05:54:10.266ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
7042024-03-29T05:54:16.836ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:64868
7052024-03-29T05:54:16.836ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 43 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:64868 req_id = 2511a3de-7c83-462b-8bcd-35a6fcff540f response_code = 200 uri = /baseboard
7062024-03-29T05:54:19.960ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
7072024-03-29T05:54:19.960ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:39871
7082024-03-29T05:54:19.968ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 8076 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:39871 req_id = c45831f4-fbec-4fcc-b2ad-e082490634cb response_code = 200 uri = /timesync
7092024-03-29T05:54:19.972ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711691605.2858346, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
7102024-03-29T05:54:19.972ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
7112024-03-29T05:54:38.758ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
7122024-03-29T05:54:38.758ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:56456
7132024-03-29T05:54:38.767ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 8972 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:56456 req_id = b8d44ee1-95a3-4411-ae17-d241a7ded8ea response_code = 200 uri = /timesync
7142024-03-29T05:54:38.769ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711691676.925229, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
7152024-03-29T05:54:38.769ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
7162024-03-29T05:54:46.867ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:62176
7172024-03-29T05:54:46.868ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 1046 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:62176 req_id = 5b4a0cfe-af48-46d2-bb29-ae19da89bb99 response_code = 200 uri = /baseboard
7182024-03-29T05:55:01.197ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
7192024-03-29T05:55:01.197ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:62323
7202024-03-29T05:55:01.206ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 8841 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:62323 req_id = 5fdb1ce2-518a-4cf2-a1b5-f79a6b580ffd response_code = 200 uri = /timesync
7212024-03-29T05:55:01.208ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711691676.925229, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
7222024-03-29T05:55:01.208ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
7232024-03-29T05:55:12.059ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
7242024-03-29T05:55:12.059ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:45503
7252024-03-29T05:55:12.068ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 9045 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:45503 req_id = 964d80b5-1779-4e47-99d3-8f9d5f2400f6 response_code = 200 uri = /timesync
7262024-03-29T05:55:12.071ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711691676.925229, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
7272024-03-29T05:55:12.071ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
7282024-03-29T05:55:16.900ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:36204
7292024-03-29T05:55:16.900ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 66 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:36204 req_id = 4e4f339b-e07f-47e3-ba0e-33d1cd8ab3ea response_code = 200 uri = /baseboard
7302024-03-29T05:55:27.589ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
7312024-03-29T05:55:27.589ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:55862
7322024-03-29T05:55:27.598ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 8611 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:55862 req_id = c679d33c-f5b1-486b-adee-551fe9d2bd53 response_code = 200 uri = /timesync
7332024-03-29T05:55:27.599ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711691676.925229, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
7342024-03-29T05:55:27.599ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
7352024-03-29T05:55:44.139ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
7362024-03-29T05:55:44.140ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:60905
7372024-03-29T05:55:44.148ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 8786 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:60905 req_id = 8cf62aa3-5603-4588-a863-b2a53c41a6ba response_code = 200 uri = /timesync
7382024-03-29T05:55:44.151ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711691742.7203174, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
7392024-03-29T05:55:44.151ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
7402024-03-29T05:55:46.932ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:52220
7412024-03-29T05:55:46.932ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 57 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:52220 req_id = 3f105bf3-3b04-44a9-8338-fe552f884dc6 response_code = 200 uri = /baseboard
7422024-03-29T05:56:05.101ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
7432024-03-29T05:56:05.102ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:46776
7442024-03-29T05:56:05.110ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 8755 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:46776 req_id = dd637db3-e9d8-4daa-a786-6d78705e46f2 response_code = 200 uri = /timesync
7452024-03-29T05:56:05.114ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711691742.7203174, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
7462024-03-29T05:56:05.114ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
7472024-03-29T05:56:16.964ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:55989
7482024-03-29T05:56:16.964ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 41 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:55989 req_id = 708abd69-ee8d-42ad-924d-3997d8adc1b8 response_code = 200 uri = /baseboard
7492024-03-29T05:56:46.996ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:34728
7502024-03-29T05:56:46.996ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 54 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:34728 req_id = e603de7a-88f1-4bac-b00f-91426ec48fd8 response_code = 200 uri = /baseboard
7512024-03-29T05:56:52.006ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
7522024-03-29T05:56:52.006ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:57068
7532024-03-29T05:56:52.018ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 11376 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:57068 req_id = 7bafe42b-11fa-4ba2-a9cc-4fd99b1da51b response_code = 200 uri = /timesync
7542024-03-29T05:56:52.020ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711691810.1358647, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
7552024-03-29T05:56:52.020ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
7562024-03-29T05:57:17.029ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:61221
7572024-03-29T05:57:17.029ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 57 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:61221 req_id = a3d55bb2-f52c-4312-8665-100a7369fdd0 response_code = 200 uri = /baseboard
7582024-03-29T05:57:24.796ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
7592024-03-29T05:57:24.796ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:57522
7602024-03-29T05:57:24.805ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 8803 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:57522 req_id = 195f195c-24d5-4922-b3cb-6c79d1bda30a response_code = 200 uri = /timesync
7612024-03-29T05:57:24.809ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711691810.1358647, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
7622024-03-29T05:57:24.809ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
7632024-03-29T05:57:47.061ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:48587
7642024-03-29T05:57:47.061ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 35 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:48587 req_id = 364e034c-0f96-499d-add3-eedad8bea404 response_code = 200 uri = /baseboard
7652024-03-29T05:57:49.794ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:801
7662024-03-29T05:58:17.095ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:200::1]:80 remote_addr = [fdb0:200::2]:55873
7672024-03-29T05:58:17.095ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 43 local_addr = [fdb0:200::1]:80 method = GET remote_addr = [fdb0:200::2]:55873 req_id = 06a32eec-0755-4ee4-a9ee-04e0bd5f9b83 response_code = 200 uri = /baseboard
7682024-03-29T05:58:17.456ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
7692024-03-29T05:58:17.456ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:58744
7702024-03-29T05:58:17.465ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 8625 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:58744 req_id = d42ed60e-70ed-437f-b84f-5ab1646e181a response_code = 200 uri = /timesync
7712024-03-29T05:58:17.467ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711691896.070551, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
7722024-03-29T05:58:17.467ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652